How to Build a Robust Continuous Integration, Delivery, and Deployment Pipeline with CircleCI and Spinnaker

Continuous integration, Continuous Delivery, and Continuous Deployment are essential practices in modern software development. This blog post covers the basics of setting up a mature CI/CD/CD pipeline for your service.

Prerequisites:

• Armory operator installed (https://github.com/armory/spinnaker-operator)
• JDK 17
• DockerHub account
• GitHub account

What’s the Difference Between Continuous Integration, Continuous Delivery, and Continuous Deployment?

Briefly, Continuous Integration(CI) is a development practice in which developers regularly integrate their code changes into the main branch of their source control system. CI helps identify problems early in the development process by automatically building and validating the artifact. If the tests fail, the changes aren’t merged. The result is better software quality, problems are caught early on, and we avoid integration challenges.

Continuous Delivery(CD), on the other hand, is the practice of making sure that the software developers are building is ready to be deployed. CD requires a robust CI, meaning continuous delivery cannot exist without continuous integration. CD is done by having automation in place to speed up the release of the new code (usually by creating an artifact that is pushed to a repository or a registry). In simpler words, with CD, we make sure that all the time we have a valid artifact ready for production deployment.

Continuous Deployment is the last step in a robust CI/CD/CD pipeline. Continuous Deployment automatically releases every code change into the production environment. Usually, Continuous Deployment is something that all companies want to do, but they are afraid of actually doing this last step. The reasons to be afraid are valid, like outages, stability, performance, and user experience issues. But with a mature Continuous Deployment process, all these issues can be non-existent.

Application

In our setup, we will use a very simple, lightweight Java application generated with Spring Boot using the https://start.spring.io page. It is a Maven application (yes, I like Maven 🙂 ), and it uses Spring Boot 3 and Java 17.

It is exposing a very simple endpoint that greets someone 🙂

@RestController
@RequestMapping("/api")
public class HelloController {

    @GetMapping(path = "/hello")
    String hello(@RequestParam String name){
        return "Hello " + name + " !";
    }
}

You can access the project by using the following Github link

Continuous Integration and Delivery Pipeline

There are many CI systems out there in the wild. But our focus will be on CircleCI, which is one of the best CI systems on the market. CircleCI has anything that an engineer would want out of a CI system:

• Supports a wide range of programming languages and tools like Java, C++, NodeJS, Docker, Python
• Source control-backed configuration
• Docker layer caching to improve the speed of the pipelines
• Integration with popular code hosting
• Integration with collaboration platforms like Slack
• Automatically deployment to various environments

In our post, we will use the free cloud-hosted version of CircleCI. Anyone can signup and take CircleCI for a ride.

The integration between CircleCI and Github is very user-friendly. You can take advantage of the one-step integration if you already have a GitHub account. On the left panel, click on Projects and search for your project

After that, you can click on Set Up Project and the next modal**,** select Fast, and click on Set Up Project

A new modal is displayed from where you need to select the Maven-based setup.

The last step would be to change the java version to 17.0.5 and click on Commit and Run.

The last step will push the .circleci/config.yml file to your GitHub project and run the pipeline.

The next obvious step would be setting up the integration with DockerHub. You can go to your DockerHub account, and you can then create a new access token. In CircleCI, set your docker user and password as environment variables in the project settings (we will reference the values in the CI pipeline)

The final CI configuration file would like below

# Use the latest 2.1 version of CircleCI pipeline process engine.
# See: <https://circleci.com/docs/2.0/configuration-reference>
version: 2.1

# Define a job to be invoked later in a workflow.
# See: <https://circleci.com/docs/2.0/configuration-reference/#jobs>
jobs:
  # Below is the definition of your job to build and test your app, you can rename and customize it as you want.
  build-test-and-push-docker:
    # These next lines define a Docker executor: <https://circleci.com/docs/2.0/executor-types/>
    # You can specify an image from Dockerhub or use one of our Convenience Images from CircleCI's Developer Hub.
    # Be sure to update the Docker image tag below to openjdk version of your application.
    # A list of available CircleCI Docker Convenience Images are available here: <https://circleci.com/developer/images/image/cimg/openjdk>
    docker:
      - image: cimg/openjdk:17.0.5
    # Add steps to the job
    # See: <https://circleci.com/docs/2.0/configuration-reference/#steps>
    steps:
      # Checkout the code as the first step.
      - checkout
      # Use mvn clean and package as the standard maven build phase
      - run:
          name: Build
          command: mvn -B -DskipTests clean package
      # Then run your tests!
      - run:
          name: Test
          command: mvn test
      - setup_remote_docker:
          version: 20.10.14
      - run:
          name: Docker login
          command: docker login -u $DOCKER_USER -p $DOCKER_PASS
      - run:
          name: Docker build
          command: docker build -t ovidiupopa07/hello-world:$CIRCLE_BUILD_NUM .
      - run:
          name: Docker push
          command: docker push ovidiupopa07/hello-world:$CIRCLE_BUILD_NUM
# Invoke jobs via workflows
# See: <https://circleci.com/docs/2.0/configuration-reference/#workflows>
workflows:
  sample: # This is the name of the workflow, feel free to change it to better match your workflow.
    # Inside the workflow, you define the jobs you want to run.
    jobs:
      - build-test-and-push-docker

Finally, the time to run the pipeline arrived. The pipeline succeeded, and the docker image was pushed to my DockerHub repository

The image is available in DockerHub.

Continuous Deployment Pipeline

Even though CircleCI supports the deployment to common targets, the integrations are less powerful than what Spinnaker provides.

The beauty and power of Spinnaker is that it can be easily integrated with different tools by using other approaches like webhooks which makes Spinnaker a very extensible platform since it can literary integrate with any tool that is able to send webhooks.

The deployment pipeline in Spinnaker requires an artifact (JAR file, Docker image, etc.). Spinnaker uses the bake stage to create an immutable image deployed to a cloud provider. Additionally, Spinnaker supports canary, blue/green, and other custom deployment strategies, but we are focusing on something else, which is the integration between CircleCI and Spinnaker.

We will create a basic Spinnaker deployment pipeline, which deploys our image to Kubernetes.

Spinnaker Installation

The first step would be to deploy Spinnaker to a Kubernetes cluster. The easiest way to deploy Spinnaker is by using Kustomize.

All the configurations can be found at https://github.com/ovidiupopa07/hello-world/tree/master/spinnaker

All you need to do is replace the value for <namespace-name>with a real namespace and to set the GitHub and DockerHub access tokens (<dockerhub_token> and <github_token>)

The next step would be to run the kubectl apply -k . command in the spinnaker folder.

After a couple of minutes, you will have your Spinnaker deployed and ready to be used

Because Spinnaker is not by default exposed to the outside world, we will need to port-forward Deck and Gate by using the following commands

kubectl port-forward svc/spin-deck 9000
kubectl port-forward svc/spin-gate 8084

When this is done, we can access Spinnaker at http://localhost:9000

Deployment Pipeline Creation

Now it’s time to create our first application. Click on the Create application button from the Applications tab and enter the app details below

After the application is created, we can move to the pipeline creation. Click on Configure pipeline and set a name

The integration between Circle and Spinnaker can be best realized by triggering changes to DockerHub. With this approach we wouldn’t have to deploy on every tag, so the CI could generate RC tags but only the fully release tags will get to production.

The artifact constraint will be defined as in the below screenshot

The pipeline definition’s final step is deploying the container in Kubernetes. I will use a Deploy Manifest stage configured like in the screenshot below. The deployment YAML file can be found at https://github.com/ovidiupopa07/hello-world/blob/master/k8s/deployManifest.yml

If we push a change in our git repository, the CircleCI pipeline will automatically start. If it succeeds, it will push a new version of the docker image to DockerHub, which the Spinnaker pipeline will automatically pick up.

When doing a push and the CircleCI pipeline finished successfully, the Spinnaker pipeline is triggered automatically, the app is deployed in Kubernetes, and a load balancer is created

When calling the load balancer endpoint in the browser, we will get the expected result

Now, let’s slightly change the welcome message and check if everything works as expected

    @GetMapping(path = "/hello")
    String hello(@RequestParam String name) {

        return "Hello again " + name + " ! Have a great day!";
    }

The pipeline is again triggered automatically, and we can see the updated message when calling the load balancer endpoint

Conclusion

Engineers and organizations need a robust solution for continuous integration, delivery, and deployment. Building a continuous deployment pipeline with CircleCI (for continuous integration and delivery) and Spinnaker for continuous deployment is a great match. CircleCI meets any requirement for building, testing, and delivering artifacts.

At the same time, Spinnaker offers great flexibility regarding deployment to different cloud providers, traffic management strategies, failover scenarios, and artifacts support. If you start with CircleCI and Spinnaker, it doesn’t mean that this two are tightly coupled. They are loosely coupled so that you can change either at any time.

CircleCI and Spinnaker are a powerful combination for a robust CI/CD pipeline, enabling engineers to deploy software confidently at any scale.