How Continuous Deployment Creates Technology Cost Savings

As our economy tightens, many companies need new ways to save money. The cost of IT, the cloud, and software development make up a significant percentage of modern organizational budgets. This blog discusses how Continuous Deployment (CD) practices can convert your spend from run-the-business (RTB) to change-the-business (CTB) spend. It explains how CD can create technology cost savings, and allow your team to accomplish more with less. It also discusses my experiences in building IT automation and cost management software, and why automation is critical for success in this economy.

Will Automation Cause Layoffs?

I first joined a technology automation company back in 2006. A common objection we heard during our sales cycle was ‘if I automate that, I’ll be out of a job’. In reality, this never happened. In fact, when the 2008 recession hit, we saw the opposite. Management viewed the teams that were automating things as more productive. They were the last to be let go—not the first.

I have never met an executive who wants to lay anyone off. Before laying off workers, many companies instead stop hiring backfills. This is what technology teams should optimize for. In this economy, assume that if one of your teammates leaves, you won’t get a replacement for them. Prepare for this eventuality by improving your team’s efficiency through automation… Don’t wait for that backfill to not come because you will no longer have the available capacity to automate. Before the layoffs, additional automation will make you more nimble. It will help you save on your technology costs, and can help you avoid the layoffs in the first place.

Making a Business Case for Continuous Deployment During a Recession

After the 2008 recession I worked on software to help CIOs manage technology spend. Historically technology spend has been a black box for the business, and it is an ever growing amount of money. CIOs need to understand how to control this spend and scale it relative to revenue. Yet, CIOs were not only looking for technology cost savings. They were also seeking ways to spend money more productively. The easiest expenses to justify are ones that directly sell products and drive revenue. The remaining costs can be broken up into two large categories: 

• RTB costs keep your services running, but don’t improve how they function.
• CTB costs focus on improving existing processes and functions to make the business more efficient.

The Importance of CTB/RTB In creating Technology Cost Savings

Many CIOs focus on changing RTB cost into CTB cost so that they spend less while delivering more. 

The ability of a technology organization to innovate can be effectively measured by this CTB/ RTB ratio. This number is the percentage of spend that is improving HOW services are delivered, as opposed to simply delivering services. Automation initiatives and Continuous Deployment initiatives are both classes of CTB spend. Building new applications is also CTB spend. However, there is a key difference between new application spend and Continuous Deployment or automation spend. New Application spend creates ongoing maintenance costs, and thus decreases the CTB/RTB ratio over time. However, Continuous Deployment and automation allow you to spend less on RTB and redirect those resources to CTB. IE, automation and Continuous Deployment are CTB spend that improves your CTB/RTB ratio while realizing overall technology cost savings.

When faced with budget cuts, CIOs are looking for ways to do more with less. As the economy tightens, growth will slow, revenue may shrink. This creates budget cuts whether you automate or not. As budgets shrink, less efficient businesses die as they lose their ability to innovate, and are thus out competed. When a budget cut happens, management is typically looking for ways to minimize the business impact of those cuts. They want to maximize the CTB/RTB ratio even when they need to shrink total spend. Automation allows for this, you just have to show your executives the opportunity and help find alternate technology cost savings.

Creating Technology Cost Savings While Continuing to Innovate

Many companies have tools they don’t need. Many also have tools that allow them to do with 1 person what would otherwise require 3 people. Before cutting spend, you need to understand the spend’s impact on the business, and whether you can redirect that spend to a better use.

Not all spend types are created equally when cutting costs. If you have already signed a contract, you may pay for a tool even if you shut it off. As such, when cutting spend, it is important to ask ‘can I repurpose the money I save’? Similarly, software tools, and headcount budgets are often determined separately. Shutting down a tool won’t let you hire a new employee, nor will layoffs let you buy new tools.

Today, Armory is going through a tool consolidation effort. For document collaboration we currently use Slab, Notion, Dropbox Paper, Google Docs and Confluence. Every single one of these costs us additional money, and we can save by consolidating. This doesn’t decrease our ability to execute, and may increase it since we’ll have fewer places to search for documents. Such an initiative is a great place to save money.

A lot of companies, including armory, do a percentage of their tool spend through cloud vendors. These contracts often have multi-year commitments in order to get discounted pricing. Cloud marketplaces give you an opportunity to shift already committed technology spend from less valuable tools to more valuable tools. As Armory shuts down unneeded tools, we are repurposing committed marketplace spend to cover new tools. This is also why Armory added our Continuous Deployment-as-a-Service offering to the AWS Marketplace.

Your most valuable tool at this time is a tool that can improve your business’s efficiency. This includes both tools to understand your spend, and tools that automate tasks so you can do more with less. Automation and CD tools both fit this bill.

What to Automate First to Accelerate Technology Cost Savings?

It depends. Do you currently know where your organization is spending its time? 35 to 50% of a distributed application’s development time often goes to manual testing and deployment processes. In many organizations a portion of this spend is unique to their company and software (e.g. product-specific test automation). However frequently a large percentage is manual processes or developing home-grown tooling that could be replaced with something off-the-shelf.

If you do not know where you are spending your time, start by adding high-level automation of your overall SDLC. A lot of teams commit code, and automatically run unit tests. Often the code then sits, waiting for manual processes. Perhaps you have a QA engineer manually testing the code. Perhaps you have an infosec org who runs a security scanner before a monthly release. These are the processes you need to identify. Once a developer has merged their code back to trunk, what still needs to happen before it reaches production? 

Initially, don’t worry about automating every step. Worry about automating the overall process flow for promoting this software all the way to production. Any time you identify a manual process that is not automated, put a manual approval in your overall automation. This will allow you to both identify and measure the performance of the manual steps you need to automate.

Manual Step Types in Your Software Release Process

Once your overall software promotion flow is automated, and can measure the manual steps along the way. You now need to decide what to do about each step. Many of these steps will be quality controls—they decrease the risk of releasing code. Their goal is to ensure that your customers remain happy. Many exist because of one bad code release that happened many years ago. Some of these steps may not matter anymore at all.

To determine if a step still matters, ask what the impact of that step is. Ask when the last time was that this step identified a critical issue. I once met n engineering department spending almost half its time on a single process… When we asked whether that process had found any bugs recently, the answer shocked me. The last critical bug it identified was two years prior. They didn’t automate that process, they shut it down, and redirected staff to running or automating other processes. Now that you know what your processes are, ask yourself if they are worth keeping around at all. If you lack the staff to automate, these steps are where you can find your first RTB technology cost savings. This first savings will let you increase your CTB spend to make further improvements.

Now that all your steps matter, your next question is what step is most worth automating.

Prioritizing Steps for Automation

I know a few reasons to automate one step before the others:

1) The step only a few people can perform.

If only few people can perform the step because it is difficult or error prone, you should automate it. There are two reasons for this. Firstly, if the individual running that step leaves, your ability to deliver software may break if it is not automated.. Secondly, such steps may be done by your most talented and capable staff. These are sometimes the staff you most need in order to build automation.

2) The highly repetitive step that that is taking the most manpower.

You are automating these steps so that you can free up resources to further improve your businesses operations. You likely have a few manual processes that eat up significantly more hours than your other processes. Automating these can free up significant labor in order to redirect to further improving your operations. Additionally, if the process is currently highly repetitive, this can make the automation effort pay off faster.

3) The step is slowing down the process the most.

The cost of a bug found after release is >4x higher to fix than if it had been uncovered during initial development. This means that simply finding bugs sooner can create technology cost savings. Context switches lower productivity, and finding issues quickly will allow them to be resolved with fewer context switches. After you have automated process types 1 and 2 far enough that you no longer need to worry about what happens when someone leaves, automate these steps. When times are good, these are the first thing to automate, when a recession looms, optimize the other two categories first.

Examples From My Career, and How CD Concepts Help Create Technology Cost Savings:

Blue/Green Deployments

At a prior company I had a software product that took a day to start when some customers upgraded. To minimize customer downtime, we manually stood up two copies of the software in parallel with that customer’s configuration. Once the new version started, we then redirect a load balancer to send traffic to the new version. Only once we were sure the customer was happy with the new version did we shut down the old version. Relatively few staff members had the skills to perform this procedure manually, making it an optimization of type one. It also was a significant bottleneck in releasing code to customers, making it a process of class 3.

Many continuous deployment tools on the market provide out-of-the-box automation for a blue/green deployment strategy. Blue/Green deployment is an off-the-shelf way of implementing the above process. We replaced the high-risk manual process that few individuals could do with an automated process that anyone can trigger. This creates an RTB technology cost savings and allows you to redirect manpower to further changing your business.

Product-specific automated tests:

I once saw a technology department that was spending almost half its engineering organization’s time on a single process. They deployed copies of customer configurations on an old and new software version, and then manually reviewed every report. For each report they would manually compare the numbers, looking for differences.

There was already a customer agnostic API for getting a list of all the reports, and downloading each report’s data….. This department wrote a tool that could be perform a diff between two versions of the software. This tool was more accurate, and it was faster, freeing up time to further automate.

The next thing automated in this process was deployment. Running this tool required standing up two environments with copies of the customer’s configuration. This was still taking a large amount of time. The solution? Add a test suite that ran after the development team’s code reached main. Have this suite automatically deploy a copy of each customer, and run the validation tool on it. This increased the organizations velocity while freeing up staff to work on further improvements. It also created technology cost savings by decreasing the overhead around labor, compute, and communication when releasing new software versions.

Many Continuous deployment software packages can help you automate flows such as this. For example, web hooks can trigger database copy scripts before deployment, or run test validation after deployment. If the validation succeeds your software release process can automatically move onto its next stage.

Security Scanners:

Many companies have compliance rules that require tools such as security scanners to be run before they deploy to production. I’ve also seen gatekeeping around such processes—a dedicated organization that must run the scanner against a dedicated environment. Before the scanner is run, someone else manually deploys a new code version to the scanning environment.

These are an opportunity for continuous deployment automation: Automatically deploy the security scanning environment, and run the security scanner. If the scanner passes, move on to the next step, otherwise cancel deployment to downstream environments.

How Continuous Deployment can Help You Create Cloud Technology Cost Savings:

When adopting tools to provide visibility into cloud spend a frequent challenge is understanding who is responsible for what infrastructure. You may know what namespace is causing most of your spend, but you need to identify who the owns it. Many cd tools provide automated tagging of the infrastructure they deploy. For example, all Armory products add annotations tracking what application deployed a given piece of configuration. When you know what app deployed it, you can easily check which users can deploy that application. Suddenly you have a standard way to find the responsible party! This can help you get technology cost savings & optimization advice back to the teams that can action it.

In recent years there are also software packages like KubeCost that make recommendations on optimizing your cloud spend. A robust Continuous Deployment pipeline can automatically deploy these recommendations, while validating that your service remains healthy following the change.

Quick Recap:

We’ve discussed ways to manage your technology spend during these tough economic times. You’ve learned how to improve your company while your budget shrinks, so you can keep up with your competitors. We’ve discussed making the business case for continuous deployment, and how CD helps your organization create technology cost savings. Is the recession increasing the risk on your business? Don’t wait, start automating your software deployment before it is too late. If you need help along your continuous deployment and automation journey let us know, we’re here to help.

image attribution