Building Trust in Your Deployment Tools

Mar 23, 2022 by Stephen Atwell

The Cloud Native Computing Foundation found an extraordinary 300 percent increase in container use between 2016 and 2020. Kubernetes’ popularity is steadily rising, with 83 percent of respondents using it for production as of mid-2020.

Kubernetes’s out-of-the-box capabilities enable organizations to accelerate delivery cycles and rapidly scale their operation, contributing to its increasing adoption. However, organizations must support the container orchestration engine with secure and efficient release and deployment workflows to get the most out of it. The CNCF 2020 report highlights container release and deployment complexity among the top challenges the respondents faced.

DevOps engineers deploy to Kubernetes (K8s) by updating associated YAML files. While it may look easy for a single deployment, it quickly becomes arduous when performing multiple deployments across various environments. These include things like development, quality assurance (QA), user acceptance testing (UAT), staging, and production.

Although Kubernetes supports rolling updates, it’s challenging to do it manually since a Kubernetes application requires multiple YAML files. Manually building these files can be error-prone because of the white space syntax.

Also, handcrafting files requires substantial knowledge of Kubernetes components. These components can complicate the overall deployment process when you need to upgrade some to get the application working.

Auditing upgrades to track new and old deployments is also challenging to do manually, as is managing rollbacks for applications comprising several deployed services. Secrets management also becomes challenging when teams work on a distributed app across various repositories. Yet, containers must be secure to be trustworthy.

Many customers also desire more advanced deployment strategies than rolling updates in order to decrease the blast radius of a given change. Leveraging advanced strategies such as Canary Deployments and blue/green deployments ensures that if a new software version has an undetected issue, it affects as few users as possible.

Some of these challenges are unique to Kubernetes, but the other, broader issues apply to successful software delivery. Adopting continuous integration and continuous deployment (CI/CD) practices can address these challenges. Using tools to automate the entire deployment process reduces Kubernetes management effort so organizations can focus on core business needs.

Let’s explore how some of Armory’s automated tools help with CI/CD and address Kubernetes scaling and security concerns.

How Spinnaker and Armory Enterprise help with Kubernetes Scaling and Security

Self-serving enterprise development teams use continuous integration and continuous deployment (CI/CD) to deliver applications faster. These CI/CD practices enable business agility to give organizations a competitive advantage. CI/CD also leaves more time for innovation within your organization.

Rolling out applications to Kubernetes requires CI/CD solutions aimed at solving deployment challenges. Several open-source solutions help automate continuous delivery pipelines to Kubernetes, like Spinnaker, ArgoCD, and Flux CD. These solutions treat Kubernetes applications as first-class citizens, deploying the applications across various Kubernetes platforms, including Google Kubernetes Engine (GKE) and Amazon Elastic Kubernetes Service (EKS).

These CI/CD tools give organizations agility and control. They can implement various deployment strategies and perform rollbacks with all the required release governance. 

Armory agent provides simplified and scalable management for all of your Kubernetes clusters when using Armory Enterprise or Open Source Spinnaker to manage your deployment pipeline.


Armory Enterprise’s pipelines as code feature allows you to maintain your Spinnaker pipelines within git, supporting a GitOps strategy. GitOps leverages Git as the single source of truth and control mechanism for creating, updating, and deleting system architecture. First, Git provides permissions control, ensuring not just anyone can make changes. Second, because Git makes all the changes, organizations can define the infrastructure immutably to prevent other changes.

Organizations Face Kubernetes Scaling Issues

Open-source CI/CD solutions work well when organizations are early adopters, with small cluster sizes and a few deployed applications. However, they aren’t well-suited to large clusters.

For one, when a cluster contains many applications, the user can experience a delayed response to their action, and deploying changes becomes painfully slow. When the service is slow to respond to invocations by Kubernetes cluster components like kubelets, registries, API servers, and more, it causes timeouts and failed deployments.

Dashboards also often fail to capture deployment changes in near real-time and display stale information. The refresh slows as the application purges all the data.

Finally, these open-source solutions are not suited to scaling individual components to handle many clusters from the centralized instance. It’s challenging to understand the working details of the open-source solution.

One open-source CI/CD solution, Spinnaker, has an enterprise version that addresses these scaling challenges. Armory Enterprise, which is built on top of Spinnaker, works on clusters of any size, addressing scalability issues through improved redundancy. Its complementary features like infrastructure dashboards, drift monitoring, role-based access control (RBAC), audits, and more fulfill the development team’s needs. Armory also offers its Agent as a standalone offering for Open Source Spinnaker that increases its ability to scale to large-scale Kubernetes infrastructure.

Open-Source Tools Trigger Security Concerns 

As open-source CI/CD pipelines automate and improve the software delivery workflow, they pose significant security risks. You need to trust that these solutions are not making your organization vulnerable to attack. Let’s examine some of these risks then see how an enterprise-level solution helps.

As enterprises prioritize software delivery speed, security and compliance are sometimes an afterthought. However, to trust their deployment tools, organizations must actively integrate security and compliance throughout their deployment pipeline. Their ability to adopt specific cloud-native tools depends on their current IT topology and organizational maturity. 

Adopting a centralized CI/CD solution presents security concerns that require rigorous monitoring, command, control, and policy mechanisms. The CI/CD solution is a single source with access to all Kubernetes objects in all clusters. It often runs with full-access service accounts that can provision new components, stop existing services, read secrets, access external components including other clusters, and more. Additionally, it persists information to perform various analyses and generate dashboards.

Enterprise-level solutions like Armory Policy Engine can help centrally enforce policy on your CI/CD pipeline to improve security and make deployment more trustworthy. Additionally Armory Agent allows central management of Kubernetes accounts, helping you avoid proliferation of credentials that can access production.

Using Armory Agent for Trustworthy Deployment

Armory Agent is a departure from the centralized CI/CD solutions. It builds upon Spinnaker’s capabilities and makes the deployment process trustworthy at a massive scale — even when you have thousands of Kubernetes clusters. 

The Agent is a lightweight, scalable service you deploy in a Kubernetes cluster. Each agent manages a single cluster so that you can scale the distributed approach for any number of clusters.

The embedded agent monitors all infrastructure and streams changes back to Spinnaker’s Clouddriver service. The distributed approach delivers several security benefits.

First, the Agent’s real-time sync captures the latest deployed changes. Also, since Spinnaker Clouddriver interacts with the agent, it decouples direct access to the Kubernetes API server, limiting a bad actor’s potential access.

Instead of centralizing account management, storing all Kubernetes accounts and credentials within Spinnaker, the Agent manages service account credentials in each cluster. This approach improves security by limiting teams to the clusters they manage.

Teams can manage permissions specifically for their clusters and quickly add new accounts without restarting Spinnaker. Finally, organizations can keep K8s clusters private with limited access since only the information Armory Enterprise requires leaves the cluster.

The Agent mitigates performance and latency issues associated with a centralized Spinnaker. It optimizes how infrastructure information is cached, making force cache refresh almost instantaneous. End-users benefit from optimal application performance, and DevOps teams benefit from optimal pipeline executions.

How Armory Project Borealis Improves Trust

Armory also offers an early access SaaS deployment offering called project borealis. It is a turnkey offering that incorporates many of our solutions from Armory Enterprise.

Project Borealis allows centralized management of Kubernetes clusters at scale. This allows you to control credential proliferation and manage all your clusters with a single toolchain. Project Borealis can also help you orchestrate your entire deployment lifecycle, across multiple environments, hosted across many different Kubernetes clusters. This helps ensure that your deployment process is consistently followed, decreasing risk and improving trust in your tools.

Project Borealis also provides advanced deployment strategies including Blue/Green Deployments  and Canary Deployments. These strategies allow you to minimize the blast radius of your deployments, so that if the software contains an undetected issue the user impact, and any negative impact on trust, are minimized. 

Project Borealis is currently available as an Early Access offering to our design partners. Spaces are still available, but filling up fast–so sign up today.


Kubernetes and cloud-native applications unlock business agility, but only when organizations support them with Kubernetes-specific CI/CD tools. Processes requiring manual intervention are error-prone and often cause bottlenecks. Organizations need automation to get the most out of their containerized services. They also need enterprise-level tools to trust that their deployments are secure.

Armory provides the secure, scalable, efficient, and reliable continuous delivery (CD) platform that you need. It builds on Spinnaker with policy-driven deployments that enable you to innovate with speed and safety.

Contact Armory today for a complimentary assessment of your software delivery practices and learn more about how your organization can benefit from safe, reliable deployments.

Share this post:

Recently Published Posts

How to Become a Site Reliability Engineer (SRE)

Jun 6, 2023

A site reliability engineer (SRE) bridges the gap between IT operations and software development. They understand coding and the overall task of keeping the system operating.  The SRE role originated to give software developers input into how teams deploy and maintain software and to improve it to increase reliability and performance. Before SREs, the software […]

Read more

Continuous Deployment KPIs

May 31, 2023

Key SDLC Performance Metrics for Engineering Leaders Engineering leaders must have an effective system in place to measure their team’s performance and ensure that they are meeting their goals. One way to do this is by monitoring Continuous Deployment Key Performance Indicators (KPIs).  CD and Automated Tests If you’re not aware, Continuous Deployment, or CD, […]

Read more

What Are the Pros and Cons of Rolling Deployments?

May 26, 2023

Rolling deployments use a software release strategy that delivers new versions of an application in phases to minimize downtime. Anyone who has lived through a failed update knows how painful it can be. If a comprehensive update fails, there are hours of downtime while it is rolled back. Even if the deployment happens after hours, […]

Read more