Skip to main content

Key SDLC Performance Metrics for VPs of Engineering

Continuous delivery done well is data-driven. Data drives decisions about what to work on and how to work on it, and it tells us whether the changes are working properly and whether things are improving. The SDLC embodies these questions. Without data, decision makers are stuck between subjective feelings and objective truths.

The leading voices in Continuous Delivery answer these questions in Accelerate by explaining the science at work in IT performance. Accelerate distills it down to four KPIs: lead time, deployment frequency, change fail percentage, and mean time to resolve. Armory’s enhanced platform powered by Spinnaker puts these KPIs into your hands. In this post, we explain how the KPIs relate to the SLDC, data available inside Armory, and future KPI integrations.

Four SDLC KPIs

Lead time is the time that passes between the beginning of development work and running in production. Shorter lead times mean faster feedback from development to production, so new ideas reach the customers faster and issues are resolved more efficiently.

Deployment frequency is how often deployments are occurring. This is a proxy measure for batch size, which is the delta between changes. Batch size is extremely difficult to measure, but it is inversely correlated to deployment frequency. The reason for this is that small batch sizes are easier to develop and deploy. So, working in smaller batches ultimately yields more deployments.

Change fail percentage captures the quality throughout the SDLC. A “change” is defined as a deployment that results in degraded service or that requires remediation (e.g., outages, hotfixes, rollbacks, or patches). Degraded service may mean missing an SLO (service level objective) or even internal targets, such as “increase conversion rate by X.”

Mean time to resolve (MTTR) is the average time that passes between the occurrence of production issues and their resolutions. Therefore, lower MTTR numbers mean that outages are shorter. MTTR also relates to lead times, since lower lead times mean that fixes deploy faster.

Lead time and deployment frequency gauge tempo, while change fail percentage and MTTR gauge quality. Together, they objectively frame performance across organizations and even across teams in the same organization. In addition, Accelerate concludes that moving the needle on these KPIs directly correlates with business success. Armory’s enhanced Spinnaker platform puts three of the aforementioned KPIs—lead time, deployment frequency, and change fail percentage—in your hands.

Lead Time

Measuring lead time requires combining two ends of the deployment pipeline: source control and the deployment process. The clock starts when an engineer begings to write code, and it stops when that code passes through the production deployment pipeline. Technically, this requires tracking commit data through all of the deployment pipeline stages.

Armory’s integrations support linking Git to deployment events. The commit date combined with a “pipeline complete” is even enough to calculate lead time. Doing the math just requires some coding. A webhook could listen to completed pipeline events, and then subtract the time from the date of the commit.

Deployment Frequency

Deployment frequency is incredibly simple to measure with Armory since it is just how often deployment pipelines run successfully. The frequency may be checked in any desired time interval, such as monthly, weekly, or daily. Deployment events contain metadata about the entire lifecycle so fine-grained measurements like deployments per day, per developer, per environment are possible.

Change Fail Percentage

Accurately measuring change fail percentage requires uniting the deployment pipeline and external monitoring systems. The deployment pipelines capture technical failures (e.g., infrastructure issues) and planned failures (e.g., canary stages). This is enough for an approximate measurement. This may be enough for some teams, but it does not account for failures while running production.

Teams can integrate systems like PagerDuty (which tracks outages and resolutions) to indicate failures in deployment event metadata. This integration requires writing up webhooks. However, this type of integration may not be necessary if canary stages are working sufficiently.

Armory’s SLA integration takes this a step further: it can track application SLOs. Missing these targets can trigger a failed deployment. In a case like this, the SLA dashboard acts as a “change success dashboard” that easily identifies which applications are performing and which are not.

Conclusion

These four KPIs are SDLC performance in a nutshell. Armory’s platform provides KPI data and the required features to integrate other systems right out of the box. Teams can take these KPIs further by integrating Armory and their existing telemetry systems for real-time monitoring. Then, they can display numbers on a dashboard and share this information with the rest of the organization. Now, the SLDC seems more objective than subjective—and this is only the beginning.

At Armory, we are committed to building our customers’ velocity. This does not stop at building software—it also involves accelerating continuous delivery with data. Continuous delivery, when used correctly, is data-driven. The “understand” layer in our product roadmap helps software teams, managers, and executives figure out what is slowing them down. Customers can identify areas with the fastest and slowest times to production, and even how their lead times are in comparison to industry peers.

Armory exposes the data that teams — and leaders– have always wanted, but have never had access to.

Accelerate tells us that the gap is increasing between the best IT performers and everyone else. This is because the best teams put effort into continuous improvements, while others do not. Successful teams continually improve their process with these KPIs. We encourage you to enhance your team’s performance with these KPIs and see the results for yourself.

These four KPIs establish your current position, but they do not give you a trajectory. It is the team’s leadership that provides the trajectory for teams to re-architect their systems and implement the changes to improve their KPIs. The leadership cultivates the practices of collaboration, communication, and goal-setting. More importantly, they create the culture that considers these KPIs in daily work.

So, where will you start? Dropping lead times? Increasing deployments? Improving quality? Or simply collecting the KPIs? The choice is yours.

May 4, 2021
|
by Nikema Prophet

What Does It Mean To Deliver Software At The #SpeedOfLife?

In this blog post, I’m excited to announce the #SpeedOfLife campaign that I will be leading for the month of May. I’m looking forward to getting to know more of the awesome people who make up our DevOps community. What is #SpeedOfLife? #SpeedOfLife is a social campaign meant to highlight users and experts in the […]

Read more

April 23, 2021
|
by Chad Tripod

Armory Enterprise install in 10 minutes

I was introduced to Armory and the open-source project Spinnaker a little over 18 months ago.  Up to that point, I’d worked with numerous open-source and proprietary software and was a specialist in application delivery and data analytics.  I immediately found out that Spinnaker was the continuous delivery platform of choice for many large scale […]

Read more

April 21, 2021
|
by Nikema Prophet

#NikemaLearns About AWS – #4 Officially AWS Certified

I’m an AWS Certified Cloud Practitioner! I took the exam and I passed. I fell behind on blog updates starting about a month ago but I studied up until the day before the test. I was somewhat disappointed in myself for not keeping my schedule but in the end, I passed and that was the […]

Read more