How to Take the Pain of Rollbacks out of Deployments

Oct 26, 2021 by Stephen Atwell

Painless deployments

Software applications have become an integral part of the business climate in most modern organizations. With an ever-increasing demand for new features and enhancement of already-existing ones, software teams constantly face novel challenges, and the pace of software development is growing by the day. To keep up with this fast-paced business climate, software teams must employ the right tools. This is where continuous delivery (CD) comes in.   

Continuous delivery is a software development practice where the software team releases sustainable changes to users securely and in real-time. This practice is crucial to how software teams deliver value, and tech companies need it to thrive in the present times. 

Continuous delivery follows a streamlined process generally referred to as the continuous delivery pipeline. At the beginning of the pipeline, the developer commits his code to the source repository. With each commit, automated tests and checks run to ensure that the code conforms to predefined standards. As soon as conformance is certified, the system automatically deploys the application to staging or user acceptance testing. Afterward, the committed code is ready for deployment on demand. 

Despite all these checks, it’s still possible for the deployed code to contain bugs or inconsistencies. The only accurate method of fully certifying that the application or a new feature works according to specification is when a client uses it. A new commit may introduce a novel feature or fix a bug but break something else in some cases. The broken piece of code often goes undetected until it gets to the final user. The code is already in production and connected to many other important components, so minimizing downtime is essential.  

Picture this scenario: One of your significant clients uses your application in the middle of the night to prepare for a critical presentation the following day. The client discovers that a feature needed to extract essential information is not working. They create an urgent support ticket, and you must attend to it immediately because millions of dollars are at stake. This isn’t the kind of situation any developer likes to be in, because it puts you and your competence in the spotlight.

A simple rollback to a previous commit should resolve the problem at least temporarily. This takes some of the pressure off you while you settle down to do a thorough investigation. But what if your tools make it difficult or impossible to roll back changes? Imagine the headache and embarrassment the downtime will cause.  

Statistics show that only one in ten developers release multiple deployments each day. For about sixty percent of developers, a new commit takes up to a week to go into production. In many cases, it takes a lot of time to troubleshoot issues and restore service when an unplanned outage occurs. And even when it takes less time, developers often consider sacrificing stability, which leads to a cycle of frequent outages when something else breaks because of a previous fix. Employing the right tools can significantly improve the speed and efficiency of the entire CD process.

Avoiding Rollbacks

While rolling back changes can present a quick fix in many cases, it’s not the most optimal option for obvious reasons. Rolling back changes means that you have lost several hours of hard work and that your new functionality is not available. This is unfavorable both for the software team and the client. But then what if you could eliminate or at least significantly reduce the need for rollbacks altogether? Of course, it will lead to a significant leap in productivity and overall efficiency of the software development life cycle. However, this can only take place when you use a reliable continuous delivery tool. A tool that helps you avoid unnecessary rollbacks and makes the process seamless whenever you use it. Armory is the perfect tool for this purpose. 

Armory, a premier member of the Continuous Delivery Foundation, offers an enterprise-grade version of Spinnaker — the world’s leading CD solution. This tool empowers developers to deploy instantly to the cloud. In addition, it enforces compliance, which fosters better security, and ensures that best practices are adhered to, minimizing the need for rollbacks. 

Armory applies a policy-driven workflow to eliminate the need for manual deployment. It integrates information between developers and security operations teams. The Armory team creates custom policies with clients by working backward from a successful deployment based on the client’s needs and critical business objectives. When the security and compliance teams agree to these policies, Spinnaker automatically enforces them, providing valuable warnings and errors to developers so they can spot and resolve issues before they go into production.  

Armory supports blue/green deployments (similar to red/black deployments). With the blue/green deployment technique, two production environments run simultaneously. One runs as a live server and the other as a test server, and you can easily interchange them. You test new features on the test server, and once certified, they are production-ready. The software team can re-route the workload from the live server to the test server. The team can switch back to the live server if they detect issues, reducing downtime to the barest minimum. 

Armory also supports canary deployments where you deploy new changes to a small subset of servers for testing first. When you certify that the changes are viable, they cascade to the rest of the servers. This way a failed deployment only affects the canary servers, limiting the downtime users experience. 

In addition to the blue/green and canary deployments, Armory offers policy enforcement, production architectural best practice enforcement, and production permissions and entitlements management. These tools employ various techniques to minimize the possibility of bugs in deployment and, in effect, the need for rollbacks altogether.

Automating Rollbacks

While developers strive to reduce the need for rollbacks to the barest minimum, there are times when it’s just the only way out — at least for temporary relief — while a development team tries to resolve an issue and restore service. When it comes down to this, employing the right tool is critical to ensure that the process is seamless and optimized for both the end-users and the software team. Armory automates rollbacks with one click, using features from Spinnaker. Spinnaker improves consistency and dependability of rollbacks enabling users to restore a previous version quickly. A version includes the main application, dependencies, and all the mechanisms required to deliver the service, such as networking and load balancing. 

Armory also supports automated Kubernetes rollbacks in Spinnaker, allowing you to roll back Kubernetes deployments based on complexity and the number of deployed artifacts. A deployment pipeline may fail for many different reasons, but you only need to do a rollback when there is a broken piece of code. This feature lets you configure your settings to guard against unwanted rollbacks by using conditional checks with Spring expressions to identify when you are deploying something broken.  

Faulty deployments and inefficient change management are at the top of the list of things that lead to downtime. Downtime can lead to huge losses for organizations to the tune of several millions of dollars and a bad reputation for the technology service provider. As much as we strive to carry out the necessary checks to reduce the frequency of bad deployments, we can’t completely rule out the possibility. Armory helps to improve the quality-of-service delivery in your business through fast-paced and efficient implementations and deployment governance. Armory offers a high level of stability and reliability, ensuring that only quality code gets to the end-user. It does this by using repeatable pipelines to deliver software, enforcing best practices in an organization.

Summary and Conclusion

This article examined what CD is. It also looked at some of the challenges of the CD process, with particular emphasis on rollbacks. The article showed that you could prevent rollbacks altogether by employing the right tools. It drew examples from some features offered by Armory like blue/green deployments that run a production and test server side-by-side and canary deployments where you deploy new releases to a small sub-set of servers first to ascertain their viability. Finally, the article posited that despite due diligence and carrying out all proper checks, it’s still possible for some broken pieces to slip through and get into production, in which case, you must roll back the deployment. 

Armory is still there to save the day with its one-click rollback features. Armory’s automated rollbacks enable users to restore a previous version quickly with consistency and dependability. This includes automated rollbacks of Kubernetes containers, which make the process painless for the development team and the end user.

We invite you to check out Armory. Contact Armory today to explore all the exciting features, request a demo, a complimentary assessment of your software delivery practices, or contact support with your inquiries.

Share this post:

Recently Published Posts

Lambda Deployment is now supported by Armory CD-as-a-Service

Nov 28, 2023

Armory simplifies serverless deployment: Armory Continuous Deployment-as-a-Service extends its robust deployment capabilities to AWS Lambda.

Read more

New Feature: Trigger Nodes and Source Context

Sep 29, 2023

The Power of Graphs for Ingesting and Acting on Complex Orchestration Logic We’ve been having deep conversations with customers and peer thought leaders about the challenges presented by executing multi-environment continuous deployment, and have developed an appreciation for the power of using visual tools such as directed acyclic graphs (DAG) to understand and share the […]

Read more

Continuous Deployments meet Continuous Communication

Sep 7, 2023

Automation and the SDLC Automating the software development life cycle has been one of the highest priorities for teams since development became a profession. We know that automation can cut down on burnout and increase efficiency, giving back time to ourselves and our teams to dig in and bust out innovative ideas. If it’s not […]

Read more