Announcing the Spinnaker Kubernetes Operator: Spinnaker Management Made Simple

Aug 3, 2020 by Stu Posluns

At Armory, we are leveraging Spinnaker to help our customers and the broader community make software delivery continuous, scalable, collaborative, and safe. We are furthering that vision today by announcing general availability (GA) of the Spinnaker Kubernetes Operator. The Operator makes managing Spinnaker, which runs in Kubernetes, dramatically simpler and more automated, while introducing new Kubernetes-native features. The Operator will make it even easier for companies around the world to adopt and scale Spinnaker and achieve continuous delivery nirvana. Let’s take a look.

What Is the Spinnaker Operator and Why Did We Build It?

The Spinnaker Operator is a Kubernetes Operator for managing Spinnaker, built by Armory and available for free. It makes deploying and managing the full lifecycle of your Spinnaker app simple, automated, and reliable, leveraging a Kubernetes-native GitOps workflow.

Kubernetes operators are application-specific controllers that leverage the Kubernetes API and deep, domain-specific expertise to automate the creation, configuration, and management of complex stateful applications. Operators have been released for popular open source projects including etcd and Prometheus.

Armory engineers have years of expertise in deploying, scaling, and managing Spinnaker in production in the world’s leading enterprises. But even with our experience, using Halyard to manage our and our customers’ Spinnaker instances was a heavier lift that we would have liked. It involved numerous manual steps entered through the CLI, lacked an easy way to store and manage configurations as code, and required engineers to ramp up on Halyard, a domain-specific tool for managing Spinnaker. Furthermore, we heard a desire from the community to be able to manage Spinnaker through the same tools they use to manage Kubernetes, without the need for context switching.

And so, we productized our years of Spinnaker expertise into the Spinnaker Operator. The Operator makes managing Spinnaker declarative, automating manual processes and enabling our teams to treat Spinnaker like just another Kubernetes deployment using kubectl and kustomize. On top of it all, the Operator enables new functionality not available in Halyard, such as pre-flight validation of config changes and the use of Kubernetes Secrets as a secrets store.

We’ve made the Operator available for free so that the entire Spinnaker community can spend more time deploying with Spinnaker and less time managing Spinnaker. Check out a quick demo here to see the Operator in action.

What Are the Main Benefits of the Operator?

Automated Spinnaker Management: Spend less time managing Spinnaker and more time innovating.
- Install Spinnaker with a few keystrokes, and upgrade your Spinnaker version simply by changing the version number in a config file and applying.
- Enable Spinnaker to deploy itself when there are changes to configs or Spinnaker is in an unhealthy state.
- Automatically expose Spinnaker to end-users via load balancers instead of detailed manual set-ups.
Kubernetes-native Tools: Kubernetes-native management of Spinnaker and access to powerful Kubernetes features.
- Instead of learning how to use Halyard, use kubectl and kustomize to install and manage Spinnaker.
- Store Spinnaker secrets in Kubernetes Secrets.
- Validate your Spinnaker configurations before applying them.
GitOps Workflows: Bring Spinnaker into your GitOps workflow by managing all configs through Git.
- Centralize your Spinnaker configurations and manage them in a single Git repo.
- Enable collaborative code reviews on config changes to ensure that Spinnaker is stable and secure.
- Built-in config version control allows rapid rollbacks and easy auditability in the event of a bad update.

“We’ve loved using the Spinnaker Operator. To begin with, the migration from Helm charts to the Spinnaker Operator was painless. Now that we have migrated, we have zero need to use Halyard scripts for advanced configurations, and the declarative syntax of the Operator is much more expressive and testable. The Operator has also taken all of the effort out of upgrading Spinnaker to the latest version – all we have to do is change one value in the CRD. The Operator webhook prevents syntax errors from bringing down productive services, and the Halyard configuration simplifies debugging and enables us to automate configuration through templating. Overall, it has been a game changer for us in how we manage Spinnaker.”

Jonathan Hatalla, Lead Infrastructure Engineer at Salesforce Einstein

Want a deeper dive on the Spinnaker Operator? Keep reading!

What Can I Use the Operator For and What Problems Does it Solve?

Installing & Upgrading Spinnaker

Installing and upgrading Spinnaker using Halyard is a bit of a pain. It involves creating and mounting a standalone Halyard machine, ensuring that the config files in the machine are up-to-date, and learning to use the Halyard CLI.

The Operator lives natively within your Kubernetes cluster – no need to mount a separate machine. Once you have the Operator installed (which takes just a few simple commands), all you need to do is have a Spinnaker config manifest and hit kubectl apply. These are regular Kubernetes manifest files, and we’ve provided a template to get started. To upgrade versions, simply change the version number in the manifest and apply – that’s it!

Updating Configs

With Halyard, config files need to be mounted within the Halyard machine and applied using hal commands. Halyard does not have a native way to manage the lifecycle of your config files, which creates numerous manual steps, operates outside of your Git workflow, and makes it difficult to track changes. There is also not an easy way to validate config changes before applying them.

With the Operator, you can have One File to Rule them All. All you have to do is make changes to the SpinnakerService.yml manifest and apply with kubectl. The manifest is stored in source control, enabling version control, auditability, and a GitOps workflow for managing Spinnaker. It also allows you to cut down on the number of individual settings you need to manage for Spinnaker’s various microservices.

For more complex use cases, you can use kustomization files to break up your configurations into smaller chunks, and even create dynamic configuration processes. With the Operator, you get the best of both worlds – simplicity by default, and more fine-grained ability to configure the manifest if you choose.

Another powerful benefit of the Operator is the ability (using Kubernetes’ validation webhook) to perform pre-flight validation on config changes. This brings additional safety and stability and helps to ensure that you don’t accidentally break your Spinnaker instance with a missing comma.

Disaster Recovery & Rollbacks

Disaster recovery with Halyard could be problematic. If the Halyard machine went down, configs may or may not be backed up, and the virtual machine needs to be remounted before you can recover Spinnaker.

With the Operator, Spinnaker manifests are stored separately, in your repo. If your cluster goes down or if you want to roll back a bad change, simply reapply the desired manifest from source control and the rest is taken care of declaratively.

How Does the Spinnaker Operator Work?

With the Spinnaker Operator all of your Spinnaker configurations are stored in native Kubernetes manifest files, as part of the Kubernetes kind SpinnakerService, defined in its own Custom Resource Definition (CRD). With this approach, you can customize, save, deploy, and generally manage Spinnaker configurations in a standard Kubernetes workflow for managing manifests. No need to learn a new CLI like Halyard, or worry about how to run that service.

The Spinnaker Operator has two flavors to choose from, depending on which Spinnaker you want to use: Open Source or Armory Spinnaker.

With the Spinnaker Operator, you can:

Use kubectl to install and configure a brand new Spinnaker instance (OSS or Armory Spinnaker).
Take over an existing Spinnaker installation deployed by Halyard and continue managing it with the Operator going forward.
Use Kustomize or Helm Charts to manage different Spinnaker installations.
Use Spinnaker profile files to provide service-specific configuration overrides (the equivalent of clouddriver-local.yml, gate-local.yml, etc.)
Use Spinnaker service settings files to tweak the way Deployment manifests for Spinnaker microservices are generated.
Use any raw files needed by configs in the SpinnakerService manifest (i.e. Packer templates, support config files, etc.)
Safely store secret-free manifests under source control, since a SpinnakerService manifest can contain references to secrets stored in S3, GCS or Vault (Vault is Armory Spinnaker only).

Additionally, Spinnaker Operator has some exclusive new features not available with other deployment methods like Halyard:

Spinnaker secrets can be read from Kubernetes secrets.
Spinnaker is automatically exposed with Kubernetes service load balancers (optional).
Generated Kubernetes manifests for Spinnaker services can be fully customized by JSON patches as part of SpinnakerService configuration.
Experimental: Accounts can be provisioned and validated individually by using a different SpinnakerAccount manifest, so that adding new accounts involves creating a new manifest instead of having everything in a single manifest.

Let’s Look at an Example Workflow

Assuming you have stored SpinnakerService manifests under source control, you have a pipeline in Spinnaker to apply these manifests automatically on source control pushes (Spinnaker deploying Spinnaker) and you want to add a new Kubernetes account:
Assuming you have stored SpinnakerService manifests under source control, you have a pipeline in Spinnaker to apply these manifests automatically on source control pushes (Spinnaker deploying Spinnaker) and you want to add a new Kubernetes account:

Save the kubeconfig file of the new account in a Kubernetes secret, in the same namespace where Spinnaker is installed.
Checkout Spinnaker config repository from source control.

Add a new Kubernetes account to the SpinnakerService manifest file, referencing the kubeconfig in the secret:

apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
  name: spinnaker
spec:
  spinnakerConfig:
    config:
      version: 2.17.1   # the version of Spinnaker to be deployed
      persistentStorage:
        persistentStoreType: s3
        s3:
          bucket: acme-spinnaker
          rootFolder: front50
+      providers:
+        kubernetes:
+          enabled: true
+          accounts:
+          - name: kube-staging
+            requiredGroupMembership: []
+            providerVersion: V2
+            permissions: {}
+            dockerRegistries: []
+            configureImagePullSecrets: true
+            cacheThreads: 1
+            namespaces: []
+            omitNamespaces: []
+            kinds: []
+            omitKinds: []
+            customResources: []
+            cachingPolicies: []
+            oAuthScopes: []
+            onlySpinnakerManaged: false
+            kubeconfigFile: encryptedFile:k8s!n:spinnaker-secrets!k:kube-staging-kubeconfig # secret name: spinnaker-secrets, secret key: kube-staging-kubeconfig
+          primaryAccount: kube-staging

Commit the changes and open a Pull Request for review.
Pull Request approved and merged.
Automatically a Spinnaker pipeline runs and applies the updated manifest.

Reduce management overhead and focus on innovating using the Spinnaker Operator!

What’s Next for the Spinnaker Operator?

The Operator is currently GA and is available open source from our repo under the Apache 2.0 license.

In the meantime, we are continuing to invest in the Operator to make installing, configuring and managing Spinnaker easier and more powerful. We’re enhancing Spinnaker iteratively, and deeply appreciate your feedback. Issues and PRs are very welcome!