Process Control Anti-Patterns that Impact DevOps

Part Two in a Series on DevOps Anti-Patterns

Introduction

So, I was thinking: what about an alternative symptom-based checklist to measure how far you are on the journey? I came up with some ideas to use as a gauge. Essentially… How DevOpsey are you?

By DevOpsey, of course, I mean all of the things in the DevOps Loop (plan, create, verify, package, release, configure, monitor). But it also applies to things like the DevOps Superpattern, and probably a bunch of other things too.

I kept adding stuff to the article until I realized that it was getting too big and needed to split it out. This second part addresses some of the anti-pattern symptoms you might come across in overly zealous process when systems thinking isn’t applied. Often these things are implemented with best intentions, just like the road to hell.

Let me know what you think.

It takes more than a few hours to go from initial commit to production

This one thing says so much about how many silos, control gates, and manual steps there are in the organization, and tools like Value Stream Mapping are there to help work through with stakeholders if you want to understand where all of that sunk time is going.

It takes more than an hour for subsequent commits to make their way to production

You shouldn’t need to wait until the end of the sprint or program increment to get your code out. You shouldn’t need to wait for the next meeting of the change control board. You shouldn’t need to wait for QA to bless the change.

https://twitter.com/jezhumble/status/1134851640182231045

Even if it takes ages to get service infrastructure stood up (see above), low latency to ripple changes to production is essential for fast feedback. In conjunction with adequate testing, short cycle times will reduce the risk of defects significantly.

Once consequence of longer commit-to-deploy cycles is that developers will bundle many commits into a single change, almost certainly increasing risk and complicating diagnosis.

Even if your organization is staunchly control-based and insists on a separate team signing off ahead of feature release, it’s possible to separate deployment (installing new code on production systems) from release (the act of making new code generally available). In this way, you can still test in production and improve confidence that new code doesn’t break existing functionality early.

You’re frequently blocked waiting on someone to perform some task or approval

Whether it’s Operations, Change Control, Database Administration, InfoSec, Compliance, QA, or IT, relying on teams who don’t have you as first priority is tough.

Empowering the organization to simplify process, create thoughtful automation, and otherwise eliminate silos will help dramatically.

Well, the NOC tested, and they said everything was fine…

In this particular case, the NOC ran some tests and said everything was fine. Unfortunately, there were no details given on what those tests were.

Because of this disempowerment, all release engineering could do is get the slave re-imaged and hope for the best. Of course, this fixed nothing. The net is that release engineering spent time dealing with complaints from developers and talking to the NOC, the NOC spent time being busy not tracking down the problem, and everyone lost.

This is another great example of how silos will hurt your organization.

Major changes require extensive cross-team coordination

Dr. Strangelove, or: How I Learned to Stop Worrying and Love the Bomb

Putting a team of cross-functional senior staff in a room to roll out a major change, perhaps by way of a “war room” or “mission control” indicates tight coupling and brittleness.

This is distinct from waiting on someone to perform work on your behalf and is more often a symptom of the complexity of application architecture than organizational complexity.

This is not the 1960s, and you are not sending rockets to the moon (although if you are, please say hello). In order to maintain feature velocity and stability, it’s important to develop components and systems that are loosely coupled and can be upgraded independently.

You observe change windows or change quiet periods

When the motivation is fear, controls are typically instituted because confidence has been lost as severe production incidents cause trauma to the business.

Stalling delivery means that unpublished changes will accumulate. This is risky, since it becomes increasingly hard to remember implementation details.

Isn’t it better to improve confidence in delivery so that small, incremental changes can be made at any time with trivial or no risk?

You have runbooks

Runbooks are often boasted about by those who insist upon them as being critical to operational success.

In practice it’s often incredibly hard to keep runbooks up to date. After all, developers like writing code and invariably detest spending time writing docs, let alone a set of operational procedures based on what might happen one day.

Even worse: for a sophisticated organization that is well on its way to maturity, the frequency of incidents should fall dramatically, and there becomes little benefit to creating and maintaining volumes of documentation “just in case”. Wouldn’t it be better to spend time developing more features and tests?

If your organization relies on runbooks, then you’re not empowering developers to maintain their own stuff and are insulating them from critical information about how their code is performing.

Next Up

https://cloud.google.com/blog/products/devops-sre/the-2019-accelerate-state-of-devops-elite-performance-productivity-and-scaling

Whether small company or large, one key observation from a summary of the 2019 State of DevOps Report is that companies which have a higher degree of control and process tend to be less performant. By the way, it also highlights how this journey is not a purely technical one.

Everyone in an organization has their role to play, but it’s incredibly disheartening to be told to work harder if you’re being hamstrung by the system. Fixing process is a foundational and critical investment.

Eighty-five percent of the reasons for failure are deficiencies in the systems and process rather than the employee. The role of management is to change the process rather than badgering individuals to do better. — W. Edwards Deming

The third part of this series will talk about software related concerns, which are mainly about how to think about writing and maintaining code. You also haven’t seen the last of W. Edwards Deming.

p.s. In case you missed it, the first part in the series covered Operations.

Software and Technology Nerd, DevOps Ninja, Maker of Things, Aerospace Enthusiast. https://orc.works/