Why Script Based Deployments to Kubernetes Don’t Scale
Apr 26, 2019 by Ethan Rogers
Jenkins has proven to be invaluable in enterprise’s DevOps tool chain for CI. It is flexible, extensible and easily coerced into doing anything that you need automated, including CD (Continuous Delivery). As adoption of Kubernetes has grown, many software shops have done what they’ve always done when new infrastructure such as AWS, OpenStack and Mesos, are introduced: write scripts, build pipelines and run them on Jenkins. While no one can deny that this solution works, everyone agrees that it doesn’t work particularly well or solve all of the problems around software delivery. Spinnaker, a purpose built platform for software delivery, aims to abstract away the underlying infrastructure, creating a platform on which to solve the problems around software delivery. Regardless of how fast AWS, GCP, Azure and others invent new infrastructure, Spinnaker’s community will keep up. A common question we hear is: Why use Spinnaker when I can just keep doing the same thing I’ve done before?
Why script based deployments don’t scale
“How do I deploy my application to Kubernetes?”When faced with this question, most teams will opt for answering this with tools that they’re already using. In most cases this is Jenkins (or another platform which allows you to run scripts or custom code). Unfortunately, these solutions only ever seem to solve a small part of the problem. While they indeed answer the question of “how” they typically don’t address questions that come after:
- How do I find which version is running in production?
- How do I rollback?
- How do I orchestrate a staged rollout?
- How do I enable teams to answer these questions without giving them keys to the kingdom?
These are all questions that need to be answered as a team or organization scales and often the answer is “write more scripts” but, time and time again teams discover that this doesn’t scale and distracts them from the real work they need to do to build business value.For Kubernetes specifically, it’s very easy to answer the question of “how”.
kubectl – a tool for interacting with the Kubernetes API – makes “deploying” manifests simple and, for a lot of teams, speeds Kubernetes adoption. However, as stated above, as adoption grows, this quickly becomes a burden when introducing multiple environment, security restrictions and compliance requirements. Let’s take a look at how Spinnaker can alleviate this pain and answer the questions that need answering.
Spinnaker vs Kubectl (Kustomize, Skaffold, etc)
Firstly, it’s important to note that the Spinnaker Kubernetes provider still uses
kubectl under the hood to deploy manifests. As far as registering resources with the Kubernetes API is concerned,
kubectl is the best tool for the job. That said,
kubectl is only a small, small piece of the software delivery puzzle as it isn’t a deployment tool and is simply an API client used to interact with the Kubernetes API. At a low level, the Kubernetes provider adds additional safety and convenience over scripted,
kubectl based deployments:
Versioned ConfigMaps and Secrets
ConfigMaps and Secrets are versioned in support of immutable, reliable deployments and rollbacks Without this feature, one-click rollbacks are not possible because rolling back a workload requires the rollback of a config change. This leaves you with two options:
- Revert the changes in git and go through the build and deploy process again.Leaving you with extended degraded services or downtime.
- Revert the change manually in the cluster. This is risky for obvious reason and becomes unscalable as the number of your Kubernetes clusters increase
For ore details on how this works, check out this blog post!
Deploy Manifest stage
The Deploy (Manifest) stage in Spinnaker encapsulates all of the logic that you have to script with
kubectl. Taking manifests from source and pushing them into the API is a small step. Once they’ve been pushed there are other questions to answer to determine if the deployment succeeded: have the resources stabilized? When should I stop trying and give up? If the resource hasn’t stabilized, why? All of these questions can be answered with
kubectl but you need even more scripts to do so.
If we zoom out a bit and look at the higher level benefits of Spinnaker we can see even more value.
Sophisticated and complex deployment strategies
While Kubernetes Deployments offer some native deployment strategies like RollingUpdate it fails to include industry standard strategies like Blue/Green or Canary. These strategies require a higher level of orchestration. Spinnaker provides these strategies out of the box, consumable without any additional orchestration. Implementing these strategies in a script based system is complex, error prone and difficult to redistribute to other teams.
Deploying into a single environment is a relatively simple task, however, the real challenge begins when you need to orchestrate deployments across multiple clusters or regions. Today’s modern production environment is made up of many environments spread across many regions or even many clouds.
Safer deployments and production management
Spinnaker provides a level of deployment & application management which hasn’t been reproduced in any other platform. Continuous deployment is a subset of software delivery as a whole and operating software once it’s been deployed is just as important as deploying the software itself. Regardless of whether or not your deployments occur with scripts or not, having the full view of your running application makes production feel like less of a black hole. It also provides a means for managing change in the event that something goes wrong. Script based deployments need script based roll backs. Since Spinnaker is responsible for managing the application’s lifecycle rollbacks become a first class citizen.
Whether you’re in operations or QA, understanding who deployed what version into an environment is critical to resolving any issue. Spinnaker provides a rich UI which will show teams the status of any deployment or pod across clusters, regions, namespace and cloud providers. This type of rich information can only be surfaced in a tool which is dedicated to the task of software delivery and management.
If you’re interested in learning more about how Spinnaker can help you deploy applications to Kubernetes reliably, take a look at a few of our other blog posts!
Recently Published Posts
Welcoming 2022: Reflecting and looking forward
Nearly all cultures globally have some form of celebration marking the Winter Solstice. Common threads found in most observances of the annual event are celebration of family and friends (living and past), reflection of the past year, and some form of giving thanks for continued health and sustenance. Exiting 2021, said celebrations would seem especially […]
Read more →
Resiliency and Load distribution
Introduction When scaling a network service, there are always two concerns: resiliency and load distribution, to understand these concepts let us first understand the broader term “Redundancy”. Redundancy is the duplication of a component to increase reliability of the system, usually in the form of a backup, fail-safe, or to improve actual system performance. Resiliency […]
Read more →
CVE-2021-44228 – log4j (Log4Shell) – an analysis
Today marked a 0-day disclosure of a rather nasty vulnerability in one of the most commonly used frameworks for logging – log4j. This one is nasty on multiple levels. Note that Armory Enterprise is NOT affected by this vulnerability. The impact on this vulnerability is likely huge and is already being exploited. Additionally it can […]
Read more →