Elegant Software Delivery Pipelines: Theory
How frequently and easily do your software developers take code from commit to production? If the answer is more than an hour, then your company is probably spending orders of magnitude more per feature delivered.
As the complexity of software delivery increases, so more people need to be involved in the process. This becomes debilitating to the business and the high performers who are driven to move the company forward.
There are a lot of things that can be done to make a difference, but one of the most fundamental is in automating how software is built and deployed.
This is the first part of a two part series detailing theory and practice of software delivery pipelines.
Everything that is described in this series is intended to help you think about driving improvements in reliability and predictability. The task of actually writing software can’t be automated, but the process of delivering software should be like a manufacturing production line, and the mechanics and operation of production lines is something that is well understood.
Let’s talk about elegance and founding principles.
But it All Started So Well…
I believe that often times, we end up with toil in the mechanism by which software is delivered because we — as organizations — focus on short-term gains through a misplaced sense of urgency, and the consequential debt accumulates over months and years.
It’s so easy to myopically focus on the here-and-now rather than invest time in building a solid foundation. Nothing is for free, and there are consequences to this approach.
The net result will be a complex system full of questionable implementation that has become vindictive towards those brave enough to use it.
As complexity increases, more mistakes are inevitably made, particularly if manual changes are used to “fix” problems at time of deployment.
As the frequency of mistakes increase (and they will), trust becomes lost with management. As trust becomes lost with management, more administrative controls are instituted in order to oversee changes which are considered “risky”.
As more administrative controls are instituted, more time is increasing spent on paperwork, in meetings, and waiting for approvals. As more time is spent on paperwork and in meetings, less is available to perform productive tasks. As more time is spent in paperwork, and approvals, the cost of change is increased, so there is pressure to batch more work. Bigger units of change ensures that each change becomes increasingly less predictable.
As productivity drops, more people are required to do a given unit of work. Adding more people increases the number of management tiers required. Steep organizations find it harder to communicate.
The cycle continues. Morale drops. The most talented individuals who can solve problems on a dime will leave the company, to be replaced by those who are change-adverse and thrive on process-heavy organization where wasted time is celebrated.
Compounded, this becomes a huge contributor to the The Eight Wastes of Lean.
People
At its most fundamental, this is about people. About overcoming cognitive bias to get individuals empowered, and minimizing their cognitive load to help them reach their full potential.
The biggest aspects of minimizing cognitive load that I’ve come across are:
- Eliminating repetitive tasks, also known as toil or yak shaving, just like Google’s SRE Book says;
- Driving consistency through approaches like preferring cattle over pets that enable huge economies of scale;
- Simplifying, streamlining, and eliminating unnecessary process and interactions
In the context of continuous delivery, the longer it takes to go from ideation to understanding how a change performs, the more expensive the change becomes. Fast feedback to developers through continuous integration and deployment with a software delivery pipeline is vital. Weekly, monthly, or quarterly releases to production do not count as continuous.
Organization
Empowerment is vital: eliminate old-fashioned organizational command and control structures. These only give the illusion of safety and efficiency while destroying innovation, morale and velocity.
Eliminate silos. Silos lead to teams wasting time waiting for others, cause errors in communication, and lead to huge sets of work in progress.
Develop teams that you trust to make good decisions in their design and execution, and are passionate about what they do. Provide tooling that makes best practices easy, and mistakes all but impossible. The word develop is vital here: you cannot trust those who do not know what to do.
Elegance
From Wikipedia, elegance is beauty that shows unusual effectiveness and simplicity.
These attributes are vital: the more complex a system is, the more unpredictable it becomes, and the more expensive it is to keep things under control.
In a practical sense, this means being uncompromising in investment to keep things simple, flexible, and avoid locking yourself into one way of thinking.
A fundamental attribute of a good architect is to ensure that a system remains flexible to change. Part of this is in deferring as many design decisions and features for as long as possible. This is as true for infrastructure and pipeline design as it is in creating software.
Beyond specific language or library, there should only be one way of obtaining a specific outcome. One set of pipeline tooling used by all, one coherent application infrastructure toolset, one way of deploying infrastructure changes in a given context, one way of monitoring/observing, and so on.
Measuring Progress
How do you know how well you’re doing, and whether you’re trending in the right direction? This article from Stackify has a really great list.
How many people are required to get a feature delivered? How many teams are required? Every time you can eliminate or consolidate a role, the faster you will go.
The best I’ve seen in a high-compliance environment is two: one to make the change, the other is someone else on the same team to approve the change. Individuals on the team switch roles frequently.
More practically, when infrastructure is immutable and deployed through configuration as code, you’ll reach the point where updates are frequent because you can deploy and roll back in minutes. This is an incredible place to be, and you will find that your infrastructure tends towards zero long-lived vulnerabilities.
Wrapping Up
At this point in time, these ideas are not rocket science, and there are a lot of companies who are leveraging them to huge advantage. If you’re not, then be careful not to be left behind in the dust, irrespective of how well things seem to be going right now.
Every company has its own set of strengths and weaknesses, so take your time, and think what’s truly right for you. The principle remains, though: the longer it takes (and the more people it takes) to reliably go from commit-to-production, the higher the cost of features delivered.
As practical ideas evolve, there will inevitably be conversations around what needs to be put into place to facilitate high productivity, and it’s easy to jump too soon into tooling.
Bear in mind that only a really, really small percentage of the time is the right answer to build it yourself. Sometimes it’s necessary to undo an old decision in order to make leaner headway, that’s okay. Reliable open source tooling is available to address all of these needs. Off-the-shelf is almost always the right approach, in spite of the protests this will provoke from those who wish to inflate their resumes or ego. I’ll get into that more next time.
Of course, continuous delivery is just half of the story, and doesn’t mean development teams can relax from relentless tiny habits in how code is built and maintained.
If you’re not sure where to start, then let me know, or check out what others were doing in the 2021 State of DevOps Report.
p.s. For related content, check these out!
