”With Armory's Platform we've reduced years of accumulated tech debt, enabling us to deploy 10x faster and significantly increase developer efficiency”
Lookout is a market leader in mobile security located in San Francisco, California. With 125 million registered mobile devices fueling a dataset of virtually all the mobile code in the world, Lookout uses machine learning to predict and stop mobile attacks before they do harm.
Lookout bases all its infrastructure on Amazon Web Services employing more than 2500 VMs for managing 120+ services responsible for securing millions of mobile users.
We talked with Brandon Leach who is the Sr. Manager of Platform Engineering at Lookout to get more details on how Spinnaker is managing deployments across the company.
Brandon spearheaded the migration to Spinnaker, succeeding in centralizing all legacy deployment practices into a single tool. Brandon was also the lead of Lookout’s transition from an on-premises datacenter to public cloud (AWS).
Spinnaker at Lookout at a glance
|Armory's Platform migration period||1 month for the first service that served as POC|
|Armory's Platform replaced||Chef, Terraform, Boto, Rundeck, Capistrano, Elastic Beanstalk|
|Engineers using Spinnaker||50|
|Applications deployed with Armory's Platform||25|
|Pipelines||Manual creation (before)/ Foremast (now)|
|Running platform||Amazon Web Services|
The pain of a legacy fragmented deployment process
Before standardizing on Spinnaker, Lookout has made many attempts at building their own custom in house deployment tooling. There were many generations of tooling being used at the same time within the company. This fragmentation was the result of an ongoing effort to build the perfect continuous delivery platform by multiple internal teams.
|Continuous delivery attempt 1||Chef/Terraform/Rundeck|
|Continuous delivery attempt 2||Docker/Terraform|
|Continuous delivery attempt 3||Gradle/Elastic Beanstalk|
|Continuous delivery attempt 4||Chef/Capistrano/Jenkins|
|Continuous delivery attempt 5||Chef/boto|
Every combination was tried with mixed level of success. It was obvious that no single tool could do what Lookout wanted and therefore most deployment methods were pieces of glue code connecting the various technologies together.
It soon became apparent that the deployment tools prevented Lookout from delivering new features with speed and quality. With so many tools the lack of standardization was a big bottleneck for future development.
- Deploying a service to production involved many manual (error prone steps) which was time consuming and error prone.
- On-boarding a new developer on how production deployments work took 3+ days (!!!)
- A lot of infrastructure code was duplicated making maintenance very difficult
The problem of multiple deployment methodologies was further exacerbated by teams abandoning tools when they were no longer “championed” by a developer (i.e. when he/she left the company). Extending the tools or accommodating them for new services became an impossible scenario as all deployment models were in different maturity states.
“Before Spinnaker we had multiple deployment solutions, cobbled together from open source tools each with its own flows and shortcomings.”
To solve the Continuous Delivery problem inside Lookout, it was decided to treat CI/CD as another product by itself. Brandon was part of a dedicated engineering team with the purpose of preparing a unified deployment model for all teams which would allow the company to innovate rather than battling with multiple tools at the same time.
The first task for fixing CD in Lookout was to find a single tool that:
- is cloud native and supports immutable infrastructure pattern
- is based on off the self software (and not custom glue code)
- has enterprise level support by a dedicated company
- is already used in production by several leading companies.
Brandon had previous experience with Asgard (the Spinnaker precursor from Netflix) making Spinnaker the natural choice of a single deployment solution within the company.
Deployments with Armory's Platform
One of the biggest advantages of Spinnaker is the ability to consolidate all existing practices into one single tool. All the features (blue/green deployments and canaries) that Lookout wanted were already offered by Armory's Platform with zero extra development.
“We used Spinnaker to replace 5 fragmented deployment methods.”
To facilitate this migration of 5 deployment models into one, Lookout created a Spinnaker adoption program with the help of Armory.
Before the Spinnaker migration started, the following success metrics were defined:
- Velocity - time it takes to deploy an application
- Throughput - number of deploys for a specified time period
- Reliability - number of defects that reach production
The first service that used Spinnaker served as the proof of concept (POC) that convinced Lookout for the merits of a unified deployment tool. Brandon started with a POC for an application that was actually used in production. Within one month the deployment solution for that service was changed to Spinnaker.
After demonstrating the results of the POC, the business green lighted the migration of more services to Spinnaker. Brandon expanded Spinnaker use expanded to 5 more services (beta phase) and Armory helped support this effort by providing day-to-day support to Lookout developers, along with extensive documentation and training resources.
“We went from nothing to a POC really fast with the help of Armory.”
To complete the Spinnaker migration, a great deal of effort was also spent integrating automated tests in the deployment pipelines. Automated tests are crucial for validating deployments and making sure that known bugs (i.e. regressions) do not reach production.
“Spinnaker is now the foundation of all CI/CD within Lookout.”
There is now a program at Lookout to migrate all services to Spinnaker by the end of Q1 2018. Most service owners are migrating to Spinnaker with little to no help. Since Spinnaker has layed the foundation for CD at Lookout Brandon’s team is now focusing on automating the creation of new services using Spinnaker.
Armory has played a crucial part in the Spinnaker migration process from day one. Armory looked at the existing methodologies used in Lookout in order to help with both the original prototype and the beta phase.
Even after the end of the POC, Armory still develops extensions for Spinnaker needed by Lookout such as automated canaries and Approved Pipelines.
Results from using Armory's Platform
The advantages of using Spinnaker were evident even after a small number of services were migrated to the new deployment paradigm.
- Number of steps to deploy in production was cut down from 25 to 1-3
- Time to deploy from staging to production was reduced from 1 hour to less than 5 minutes
- Training needed for a new developer to deploy into production was now 30 minutes instead of 3 days.
The most important metric is of course the attitude of developers when it comes to deployments. What used to be a big pain point for Lookout is now a standardized unified solution instead of a spaghetti of custom glue code.
“In the two-and-a-half years, I've been at Lookout, deploying services has always been a significant challenge. Spinnaker has, far and away, provided the smoothest experience yet.”
The combination of the Spinnaker web interface with the automatic creation of deployment pipelines makes it easy for Lookout developers to operate at cloud speed without the technical issues of the past.
Next Steps with Armory's Platform
As Lookout’s Engineering team has now standardized on Spinnaker, they are looking to advance to Stage 5 of the software delivery evolution and further reduce time it takes to deliver business value. Working with Armory, Lookout is now advancing towards continuous deployment by leveraging Armory's Platform’s advanced features, including:
- Barometer: http://docs.armory.io/user-guides/barometer/
- Certified Pipelines: http://blog.armory.io/introducing-armory-certified-pipelines/
- SLA Dashboard: http://blog.armory.io/enabling-sla-measurement-with-armory-spinnaker/
- Automated Load Testing: http://blog.armory.io/automate-load-testing-armory-spinnaker/
Armory helps software teams ship better software, faster. The Core of our Platform is powered by Armory's enterprise distribution of Spinnaker, the open-source continuous delivery platform created by Netflix and Google. Armory's Platform enables you to increase your company’s safety and velocity, decrease your “time to value”, make your engineering teams more efficient, and minimize the blockers that slow your application teams down.