Key SDLC Performance Metrics for VPs of Engineering

Continuous delivery done well is data-driven. Data drives decisions about what to work on and how to work on it, and it tells us whether the changes are working properly and whether things are improving. The SDLC embodies these questions. Without data, decision makers are stuck between subjective feelings and objective truths.

The leading voices in Continuous Delivery answer these questions in Accelerate by explaining the science at work in IT performance. Accelerate distills it down to four KPIs: lead time, deployment frequency, change fail percentage, and mean time to resolve. Armory’s enhanced platform powered by Spinnaker puts these KPIs into your hands. In this post, we explain how the KPIs relate to the SLDC, data available inside Armory, and future KPI integrations.

Four SDLC KPIs

Lead time is the time that passes between the beginning of development work and running in production. Shorter lead times mean faster feedback from development to production, so new ideas reach the customers faster and issues are resolved more efficiently.

Deployment frequency is how often deployments are occurring. This is a proxy measure for batch size, which is the delta between changes. Batch size is extremely difficult to measure, but it is inversely correlated to deployment frequency. The reason for this is that small batch sizes are easier to develop and deploy. So, working in smaller batches ultimately yields more deployments.

Change fail percentage captures the quality throughout the SDLC. A “change” is defined as a deployment that results in degraded service or that requires remediation (e.g., outages, hotfixes, rollbacks, or patches). Degraded service may mean missing an SLO (service level objective) or even internal targets, such as “increase conversion rate by X.”

Mean time to resolve (MTTR) is the average time that passes between the occurrence of production issues and their resolutions. Therefore, lower MTTR numbers mean that outages are shorter. MTTR also relates to lead times, since lower lead times mean that fixes deploy faster.

Lead time and deployment frequency gauge tempo, while change fail percentage and MTTR gauge quality. Together, they objectively frame performance across organizations and even across teams in the same organization. In addition, Accelerate concludes that moving the needle on these KPIs directly correlates with business success. Armory’s enhanced Spinnaker platform puts three of the aforementioned KPIs—lead time, deployment frequency, and change fail percentage—in your hands.

Lead Time

Measuring lead time requires combining two ends of the deployment pipeline: source control and the deployment process. The clock starts when an engineer begings to write code, and it stops when that code passes through the production deployment pipeline. Technically, this requires tracking commit data through all of the deployment pipeline stages.

Armory’s integrations support linking Git to deployment events. The commit date combined with a “pipeline complete” is even enough to calculate lead time. Doing the math just requires some coding. A webhook could listen to completed pipeline events, and then subtract the time from the date of the commit.

Deployment Frequency

Deployment frequency is incredibly simple to measure with Armory since it is just how often deployment pipelines run successfully. The frequency may be checked in any desired time interval, such as monthly, weekly, or daily. Deployment events contain metadata about the entire lifecycle so fine-grained measurements like deployments per day, per developer, per environment are possible.

Change Fail Percentage

Accurately measuring change fail percentage requires uniting the deployment pipeline and external monitoring systems. The deployment pipelines capture technical failures (e.g., infrastructure issues) and planned failures (e.g., canary stages). This is enough for an approximate measurement. This may be enough for some teams, but it does not account for failures while running production.

Teams can integrate systems like PagerDuty (which tracks outages and resolutions) to indicate failures in deployment event metadata. This integration requires writing up webhooks. However, this type of integration may not be necessary if canary stages are working sufficiently.

Armory’s SLA integration takes this a step further: it can track application SLOs. Missing these targets can trigger a failed deployment. In a case like this, the SLA dashboard acts as a “change success dashboard” that easily identifies which applications are performing and which are not.

Conclusion

These four KPIs are SDLC performance in a nutshell. Armory’s platform provides KPI data and the required features to integrate other systems right out of the box. Teams can take these KPIs further by integrating Armory and their existing telemetry systems for real-time monitoring. Then, they can display numbers on a dashboard and share this information with the rest of the organization. Now, the SLDC seems more objective than subjective—and this is only the beginning.

At Armory, we are committed to building our customers’ velocity. This does not stop at building software—it also involves accelerating continuous delivery with data. Continuous delivery, when used correctly, is data-driven. The “understand” layer in our product roadmap helps software teams, managers, and executives figure out what is slowing them down. Customers can identify areas with the fastest and slowest times to production, and even how their lead times are in comparison to industry peers.

Armory exposes the data that teams — and leaders– have always wanted, but have never had access to.

Accelerate tells us that the gap is increasing between the best IT performers and everyone else. This is because the best teams put effort into continuous improvements, while others do not. Successful teams continually improve their process with these KPIs. We encourage you to enhance your team’s performance with these KPIs and see the results for yourself.

These four KPIs establish your current position, but they do not give you a trajectory. It is the team’s leadership that provides the trajectory for teams to re-architect their systems and implement the changes to improve their KPIs. The leadership cultivates the practices of collaboration, communication, and goal-setting. More importantly, they create the culture that considers these KPIs in daily work.

So, where will you start? Dropping lead times? Increasing deployments? Improving quality? Or simply collecting the KPIs? The choice is yours.