True Continuous Deployment: From Dream to Reality with Spinnaker

At Upside, we take pride in our deployment culture. We ship new versions of code multiple times per day through our various environments powered by AWS and Kubernetes. We run an infrastructure stack that leverages the powerful microservice architecture pattern that sees our applications broken up into smaller segments that each do one thing, but do it very well. Thanks to our applications being loosely coupled, we feel safe deploying fast and deploying often due to blast radius being minimized.

Half the battle of implementing true CI/CD is already won at Upside due to our awesome engineering management and teams being fully supportive of the culture. The other half of the battle is implementing it from a technical standpoint. A big portion of this effort involves weighing the pros and cons of the many tools out there in regards to your specific CI/CD needs. This is very important; just because there is a shiny new item out there, doesn’t necessarily mean that it is a one size fits all for everyone. To quote the late American psychologist Abraham Maslow, “If all you have is a hammer, everything looks like a nail”.

CI/CD Overview

Let’s take a step back to fully flesh out what CI/CD means, because it can help us start to visualize the problem that we believe Spinnaker helps us to solve.

CI/CD stands for “Continuous Integration and Continuous Delivery/Deployment”:

Continuous Integration

Continuous Integration deals with the culture of integrating code into a shared repository multiple times per day. Often times, automated tests are run on every merge, and artifact builds are kicked off. In our case at Upside, upon every merge to master, we run tests and then build and push a new Docker image that represents that version of the code.

Continuous Delivery/Deployment

These are actually two different cultures and the difference between the two can really elevate a team’s ability to roll new services out. Continuous Delivery deals with automating the process of deploying artifacts into your environments. Typically this involves an automated deployment into dev, in which some sort of a manual judgement is needed to then deploy that artifact to later environments, and then finally ending in a deployment to production. Continuous Deployment takes this a step further and removes these manual judgements. This can be a heavy undertaking due to engineers having to write and rely on automated testing which will dictate if the code should be promoted to the next environment or not. Another roadblock to Continuous Deployment is finding the right tooling that balances out-of-the-box functionality but still leaving enough room for customization. At Upside, we are making the push to get to full Continuous Deployment, and Spinnaker is what is enabling us to get there.

Spinnaker!

Now that we can start to see what the true nature of Continuous Deployment looks like, let’s talk about the open-sourced tool developed by Netflix called Spinnaker that helps to enable it.

Also, a disclaimer that this article is less of a “how to use Spinnaker”, and more of a high level “how we are leveraging Spinnaker at Upside”. If you are interested in how to get started, check out the Spinnaker docs on how to install it and create your first pipeline.

Okay, I understand the problem — but what exactly is Spinnaker?

In a nutshell, Spinnaker is an open-sourced and self-hosted platform that allows you to quickly build out deployment pipelines. It is also multi-cloud in that you can easily deploy with built in functions to AWS, Azure, GCP, and Kubernetes to name a few. For example, need a stage in your pipeline to deploy a Helm chart to a Kubernetes cluster? Easy, there’s a built in function for this. Oh, but it needs to be a Canary deployment and analysis? Done, Spinnaker comes with that built in functionality as well. At Upside, these built in functionalities are a big reason of what sold us on Spinnaker — could you imagine scripting these in Jenkins?

A few Spinnaker terms

• Application — an application can have 0 to many pipelines. In our case, each one of our microservices is its own application in Spinnaker.
• Pipeline — a pipeline has 0 to many stages. At Upside, each of our Spinnaker applications has a pipeline that deploys to our dev environment, another that deploys to our staging environment, and finally one for our prod environment.
• Stage — represents an action in a pipeline. For example, a stage to deploy a Helm chart, another stage to publish this deployment information to our centralized logging solution, and a stage that may require a manual approval to continue the pipeline, to name a few.
• Pipeline-template — this is a parameterized JSON file that represents a pipeline. This is where we can define a pipeline and its stages in code. The purpose of this is for reuse. For example, at Upside, we deploy most of our 200+ microservices in the same fashion, therefore we want a single source of truth when we define our pipelines. So when the time comes to add an application to Spinnaker, we just create its dev, stage, and prod pipelines and point each to its respective pipeline-template. Another benefit of this is if, for example, a new stage needs to be added to all of our applications’ dev pipelines, we just edit the dev pipeline-template, and the change propagates to all of our applications.

Let’s take a look at an example

Above is a somewhat simple deployment pipeline that we use here at Upside, but still nonetheless complex. I won’t get into the specifics of what each stage is doing, but the pipeline is triggered either manually in the Spinnaker UI, or triggered on a new image hitting a Docker registry. The pipeline then uses Helm to bundle our Kubernetes spec files into an artifact that can be used downstream by our Deploy stage.

Then we have a manual approval stage that is dynamically enabled if our Run Tests stage sees something in our newly deployed resources that would need human intervention for. This manual approval has 3 options:

1. Stop — this stops the pipeline and the next environment’s pipeline will not execute.
2. Stop and Rollback — this option is selected via the dropdown in the image below and will stop the deployment from going to the next environment, as well as also rolling back the current environment’s deployment to the previous image version.
3. Continue — this continues the deployment to the next environment.

Why this deployment flow is an improvement

Previously at Upside, we had continuous deployment to our Dev environment in that upon merge to master, Jenkins would use Helm to deploy to Dev. It was then up to the engineering teams to trigger this Jenkins job to promote that to our Staging environment, and then finally to Prod. Spinnaker helps improve upon this in the following ways:

• More complexity and robustness in our pipelines flows. We began to hit a ceiling with Jenkins in that scripting any further complexity was going to become very tedious. Spinnaker enables us to quickly roll out more robust pipeline workflows.
• A single pane of glass. The Spinnaker UI allows diving into various bits of the infrastructure that each pipeline deploys such as pod logs, ReplicaSet information, the ability to scale up and down pods, etc. Also, engineers are able to make manual judgements to continue/stop/rollback within the Spinnaker UI.
• Engineers dictate their own Continuous Deployment fate. At Upside, we allow our engineering teams the freedom to write their own tests that dictate whether or not to have full automation to Prod with no manual approvals needed (aka Continuous Deployment). If their self-defined tests pass, then the deployment will automatically continue to the next environment. Spinnaker helps to enable this due to the ease of dictating pipeline branching with conditionals.
• Much more robust built in functionality for deployment strategies such as Dark, Red/Black, Highlander, and Canary. Again, for emphasis — could you imagine scripting all of this in Jenkins? For us, this is what sold us on Spinnaker.

Final Takeaways

The Good

• Lots of built in functionality — this leads to being able to quickly create more robust and complex pipelines.
• Reusability — with our microservice architecture where most services are usually deployed in the same fashion, pipeline-templates allow us to propagate the same pipeline workflow amongst all of our Spinnaker applications.
• Visibility — the Spinnaker UI offers a lot of information about a pipeline execution as well as its deployed Kubernetes infrastructure.
• Pipeline workflow — Spinnaker makes it a breeze to have stages in a pipeline rely on other stages, run in parallel, or be enabled/disabled upon certain conditionals, to name a few.
• Lots of provider plugins — While we are only taking advantage of the Kubernetes provider, Spinnaker has a plethora of other provider plugins to run deployments to a multitude of platforms (AWS, GCP, Azure, Oracle, etc.)

The Bad

• Less documentation and information sources — this comes with any cutting-edge technology that is still fairly new, but there is not much out there on Spinnaker aside from the documentation and the official Slack channel.
• Confusing error messages — there are some known bugs that we stumbled upon that lead to confusing and misleading error messages when writing out our pipeline-templates.

All in all…

Spinnaker has given our team more runway to build out complex and robust deployment pipelines with ease. With more robust deployment pipelines, we are able to not inch, but begin running towards our goal of true Continuous Deployment. With Continuous Deployment, we can further increase not only the velocity of our deployments, but also our confidence in them.

Interested in a fast-paced and cutting-edge development and deployment culture? Check out our open positions here at Upside. Also feel free to shoot Jeremy an email about Spinnaker, his SRE experience, or really anything in general!

[email protected]