How Armory and Spinnaker Support High Performing Teams (part 3)

Jan 12, 2019 by Armory

The “Accelerate” book by Nicole Forsgren, Jez Humble and Gene Kim identifies 24 key practices and capabilities that characterize high performing teams and organizations. These are the direct result of the State of DevOps research that the authors have led over the past 6 years.

The capabilities are grouped into 5 dimensions:

In previous posts, we highlighted how Spinnaker helps support “version control (everything-as-code)”, “trunk-based development”, “loosely coupled architectures”, and “deployment automation” capabilities.

In this post, we will discuss how Spinnaker helps high performing teams practicing:

Team experimentation

“If a development team isn’t allowed, without authorization from some outside body, to change requirements or specifications in response to what they discover, their ability to innovate is sharply inhibited.”
(chapter 8)

Spinnaker was designed as a Continuous Delivery platform for automating the deployment of software changes at high velocity. This means that changes going live are typically small, with a very clear scope – typically at sub-feature level. This allows a fine-grained level of control and experimentation over what gets released.

Teams using Spinnaker often run the CI part (build + unit tests) of their delivery pipeline in a CI tool like Jenkins or TravisCI, then kickoff a Spinnaker pipeline to run other activities like system or performance testing, environment promotion and so on, until finally deploying to production.

That said, it is possible to include role-based manual gates in a Spinnaker pipeline, thus reducing a team’s autonomy to experiment by depending on the approval from someone outside the team. However, these gates are being used more and more sparsely, as organizations understand that blocking a pipeline leads to an accumulation of changes (batching) waiting to be released, increasing the risk of problems and time to diagnose them.

Shifting from pipeline execution to pipeline design approval with certified pipelines

Leading organizations working in industries with strong compliance and risk control requirements are replacing role-based manual approvals with automated policies and criteria embedded in their delivery pipelines. Changes failing to comply still don’t progress to production, but there is no waiting time involved. Feedback is as fast as possible and teams start to modify their default behavior by baking in security and quality at early stages of development. They know that otherwise they are shooting themselves in the foot as changes are not really done and will require more rework.

Armory has effectively codified this recommendable practice in a feature called certified pipelines. We can see this as shifting the approval process from the pipeline execution to the pipeline design. Whoever is responsible for ensuring compliance and risk control (either a Change Approval Board – CAB – or a dedicated team) can review the policies and checks in the pipeline and “certify” it as fit for deploying changes to production. When any of the steps in the pipeline is modified, it needs to be “re-certified” again to be able to continue deploying.

This drastically reduces the required manual effort to keep a smooth flow of changes. Consider that a pipeline design might change weekly at most, so it would fit well with a CAB’s typical meeting schedule. Pipelines execute daily at a minimum, and up to hundreds or thousands of times per day for high performing teams. The overwhelming majority of them would never get past a manual approval gate, even if the change matches all the criteria for deployment (for example, passing all tests, no vulnerabilities detected, no alarms on code analysis, no performance degradation, etc).

Controlling visibility of changes and leveraging data insights for release decisions

Once we get beyond manual approval bottlenecks, how can a team effectively experiment and trial changes without risking propagating either technical failures (they will happen eventually, even if our robust test suites found no issues) or behavioral failures (i.e. the clients didn’t react to the software changes as we hypothesized) to all their customers?

Canary releases come to the rescue:

“Canary release is a technique to reduce the risk of introducing a new software version in production by slowly rolling out the change to a small subset of users before rolling it out to the entire infrastructure and making it available to everybody.”

Spinnaker supports this type of slow release of changes, exposing them to a (configurable) portion of live traffic. The goal is to analyze the impact of those changes on a small subset of users in order to decide if it should be made available to everyone or instead rolled back.

There are multiple possible factors to analyse in a canary, from infrastructure usage and run-time performance (especially for technical issues) to internal application metrics and real user monitoring (especially for behavioral impact on user). Different changes might require different combinations of data analysis.

Spinnaker can help take a step further here by integrating with Kayenta, a tool developed by Netflix to support fast and safe delivery. Its “Automatic Canary Analysis” (ACA) feature essentially takes infrastructure and application metrics from a (configurable) set of sources on one hand, and a pre-defined set of quality criteria in order to automatically make a roll out or rollback decision for a particular canary.

It’s also possible to define under which criteria a manual decision should be made based on how the canary scored. For example, we can configure that a score between 0 and 70 means the canary has failed and is automatically rolled back (all traffic directed to the previous software version again). Between 70 and 85 it’s doubtful and requires manual intervention. Between 85 and 100 the canary gets rolled out automatically (all traffic will eventually get redirected to the new version, according to the chosen deployment strategy).

For details on the rationale behind the tool and the decision computation algorithm see Netflix’s blog post introducing Kayenta.

Netflix’s blog says that “having detailed insight into why a canary release failed is crucial in building confidence in the system”. We would add to that “and enabling low risk team experimentation”.

Customer feedback

Direct feedback option in Microsoft Office (source:
Share this post:

Recently Published Posts

How to Become a Site Reliability Engineer (SRE)

Jun 6, 2023

A site reliability engineer (SRE) bridges the gap between IT operations and software development. They understand coding and the overall task of keeping the system operating.  The SRE role originated to give software developers input into how teams deploy and maintain software and to improve it to increase reliability and performance. Before SREs, the software […]

Read more

Continuous Deployment KPIs

May 31, 2023

Key SDLC Performance Metrics for Engineering Leaders Engineering leaders must have an effective system in place to measure their team’s performance and ensure that they are meeting their goals. One way to do this is by monitoring Continuous Deployment Key Performance Indicators (KPIs).  CD and Automated Tests If you’re not aware, Continuous Deployment, or CD, […]

Read more

What Are the Pros and Cons of Rolling Deployments?

May 26, 2023

Rolling deployments use a software release strategy that delivers new versions of an application in phases to minimize downtime. Anyone who has lived through a failed update knows how painful it can be. If a comprehensive update fails, there are hours of downtime while it is rolled back. Even if the deployment happens after hours, […]

Read more