The How and Why of Templated Pipelines

In an enterprise setting, it’s likely you’ll have not just more than one pipeline—you’ll have thousands. Netflix, for example, has over 9,000 individual pipelines. Sometimes a single individual is responsible for managing hundreds of pipelines.

Keeping all of those pipelines up-to-date, ensuring configurations are consistently applied and making any change in the pipeline could become a full-time job that doesn’t leave room for anything else. They also make it possible to take advantage of the fact that many pipelines are very similar, if not identical. Reusing pipelines through templates makes sense, saves time and reduces the number of errors that come with configuring a pipeline from scratch every time and/or manually updating or changing hundreds of pipelines.

We talk a lot about ‘paved road,’ or workflows that make it easy for deployments be rolled out in a repeatable, safe way. Having a standard way to manage deployments, one that works for most teams most of the time, makes it easier to ensure the entire organization is following deployment best practices, regardless of the project or the deployment target.

Using pipeline templates also helps manage multi-cloud deployments or even Kubernetes deployments on top of a single cloud. Templates can and should be target-agnostic, and the specifics for each deployment target can be added into the template’s child pipeline.

Once deployment pipelines are standardized using templating, it also becomes possible to manage infrastructure changes through the Spinnaker pipeline—including running testing and canary analysis on resources such as changes in instance types, firewall rules and load balancers. Applying the same process to infrastructure changes that is applied to application changes makes those changes safer.

How does it work?

If you’re using Armory’s Spinnaker installation, you can use Armory Dinghy service. Pipelines can either be created declaratively or can also be created from an existing pipeline.

In a standard Spinnaker installation, you can use the Spinnaker Managed Pipeline Templates feature.

The exact process and file naming conventions differ slightly between the two services. However, in general the steps are:

• Create a new pipeline or identify an existing pipeline you’d like to use as a template
• Decide and define which parts of the pipeline should be variables that can be changed on the child templates. Remember that this pipeline template should remain cloud-provider agnostic, so environment-specific variables should be set at each child pipeline
• Create the template, either by creating a dinghyfile or pipeline-template file

Once you have a template, you can create child pipelines from that template.

If you modify the parent template, a GitHub or BitBucket webhook will find and update all of that template’s dependent pipelines. All of the dependent templates are automatically updated whenever the template is updated.

Using pipeline templates makes it easier to manage the complexity of thousands of microservices. It creates a paved road that makes handling the deployment process consistently across an organization and across cloud environments possible.

Templates do not necessarily take flexibility away from any part of the team. The child templates can still be altered to add additional stages, to increase wait times between stages or in other ways to meet the needs identified for a specific application. However, cutting down ad-hoc pipeline creation not only saves time but contributes to an overall safer, more repeatable deployment process.