The How and Why of Templated Pipelines
Mar 28, 2019 by Armory
In an enterprise setting, it’s likely you’ll have not just more than one pipeline—you’ll have thousands. Netflix, for example, has over 9,000 individual pipelines. Sometimes a single individual is responsible for managing hundreds of pipelines.
Keeping all of those pipelines up-to-date, ensuring configurations are consistently applied and making any change in the pipeline could become a full-time job that doesn’t leave room for anything else. They also make it possible to take advantage of the fact that many pipelines are very similar, if not identical. Reusing pipelines through templates makes sense, saves time and reduces the number of errors that come with configuring a pipeline from scratch every time and/or manually updating or changing hundreds of pipelines.
We talk a lot about ‘paved road,’ or workflows that make it easy for deployments be rolled out in a repeatable, safe way. Having a standard way to manage deployments, one that works for most teams most of the time, makes it easier to ensure the entire organization is following deployment best practices, regardless of the project or the deployment target.
Using pipeline templates also helps manage multi-cloud deployments or even Kubernetes deployments on top of a single cloud. Templates can and should be target-agnostic, and the specifics for each deployment target can be added into the template’s child pipeline.
Once deployment pipelines are standardized using templating, it also becomes possible to manage infrastructure changes through the Spinnaker pipeline—including running testing and canary analysis on resources such as changes in instance types, firewall rules and load balancers. Applying the same process to infrastructure changes that is applied to application changes makes those changes safer.
How does it work?
If you’re using Armory’s Spinnaker installation, you can use Armory Dinghy service. Pipelines can either be created declaratively or can also be created from an existing pipeline.
In a standard Spinnaker installation, you can use the Spinnaker Managed Pipeline Templates feature.
The exact process and file naming conventions differ slightly between the two services. However, in general the steps are:
- Create a new pipeline or identify an existing pipeline you’d like to use as a template
- Decide and define which parts of the pipeline should be variables that can be changed on the child templates. Remember that this pipeline template should remain cloud-provider agnostic, so environment-specific variables should be set at each child pipeline
- Create the template, either by creating a dinghyfile or pipeline-template file
Once you have a template, you can create child pipelines from that template.
If you modify the parent template, a GitHub or BitBucket webhook will find and update all of that template’s dependent pipelines. All of the dependent templates are automatically updated whenever the template is updated.
Using pipeline templates makes it easier to manage the complexity of thousands of microservices. It creates a paved road that makes handling the deployment process consistently across an organization and across cloud environments possible.
Templates do not necessarily take flexibility away from any part of the team. The child templates can still be altered to add additional stages, to increase wait times between stages or in other ways to meet the needs identified for a specific application. However, cutting down ad-hoc pipeline creation not only saves time but contributes to an overall safer, more repeatable deployment process.
Recently Published Posts
Monitoring Spinnaker: Part 1
Overview One of the questions that comes up a lot is how you monitor Spinnaker itself. Not the apps Spinnaker is deploying, but Spinnaker itself and how it’s performing. This is a question that has a lot of different answers. There are a few guidelines, but many of the answers are the same as how […]
Read more →
The Importance of Patents: Interview with Nick Petrella, Head of Legal
In honor of Armory’s recent acquisition of a patent for continuous software deployment, we sat down with Nick Petrella, Head of Legal, for a casual conversation covering a wide range of subjects, from patent law to Nick’s background as a software engineer and why he made the leap to the law. Check out […]
Read more →
Armory Agent for Kubernetes Simplifies K8s Complexity
I’ll be upfront with you, I’m a sucker for a good origin story. It’s one of the reasons I spent hours engrossed in the Marvel Cinematic Universe not too long ago. Rooting for incredibly flawed individuals with an outsized sense of duty and superpowers to back it up….What’s not to love? My partner has a […]
Read more →