← Back to posts

Fine Grained Rate Limits for Spinnaker Clouddriver

Feb 7, 2017 by Isaac Mosquera

When using Spinnaker, it queries your cloud provider (AWS, GCP, Azure, Kubernetes, etc) frequently to understand about the state of your existing infrastructure and current deployments. In doing so however, you might run into rate limits imposed by the cloud provider. On AWS you might see an exception similar to the following:

com.amazonaws.AmazonServiceException: CloudWatchAlarm Rate exceeded (Service: AmazonAutoScaling; Status Code: 400; Error Code: Throttling; Request ID: a217a0a2-da7e-11e5-734a-a1917861e2d6)
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1160)
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:748)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:467)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:302)
at com.amazonaws.services.autoscaling.AmazonAutoScalingClient.invoke(AmazonAutoScalingClient.java:3246)

This typically means you have hit the rate limits for the cloud provider.

Recently, the Spinnaker community has added service limits as part of Clouddriver to address growing concerns about these limits. Below is an example configuration for global rate limits for all services that you would place in clouddriver-local.yml.

serviceLimits:
  defaults:
    rateLimit: 20

If you have multiple cloud providers, you can limit each one differently:

serviceLimits:
  cloudProviderOverrides:
    aws:
      rateLimit: 15

You can provide account specific overrides as well in case you have significantly more resources in one account while others have less

serviceLimits:
  accountOverrides:
    test:
      rateLimit: 5
    prod:
      rateLimit: 100

And lastly, you can have more fine-grained control for particular AWS endpoints that might have a different rate limits:

serviceLimits:
  implementationLimits:
    AmazonEC2:
      defaults:
        rateLimit: 200
      accountOverrides:
        prod:
          rateLimit: 500
    AmazonElasticLoadBalancing:
      defaults:
        rateLimit: 10

Using these rate limits will help you avoid hitting the rate limits and potentially make Spinnaker more responsive as the cloud provider clients won’t have to implement back-off strategy to continue to query the infrastructure.

To see more details and other implementationLimits, visit https://docs.armory.io/docs/armory-admin/rate-limit/#fine-grained-rate-limits

As always, we want to provide ridiculously responsive support, let us know how we can help.

Learn More

Recently Published Posts

Reduce the Blast Radius of a Bad Deployment with Automated Canary Analysis

May 23, 2022

Software deployment processes differ across organizations, teams, and applications. The most basic, and perhaps the riskiest, is the “big bang deployment.” This strategy updates all nodes within the target environment simultaneously with the new software version. This deployment strategy causes many issues, including potential downtime or other issues while the update is in progress. It […]

Read more →

Reliable and Automatic Multi-Target Deployments

May 16, 2022

Multi-target deployments can feel tedious as you deploy the same code over and over to multiple clouds and environments — and none of them in the same way. With an automatic multi-target deployment tool, on the other hand, you do the work once and deliver your code everywhere it needs to be. Armory provides an […]

Read more →

Learning out Loud: KubeCon EU edition

May 11, 2022

KubeCon+CloudNativeCon EU is one of the world’s largest tech conferences. Here, users, developers, and companies who have and intend to adopt the Cloud Native standard of running applications with Kubernetes in their organizations come together for 5 days. From May 16-20, 2022, tech enthusiasts will congregate both virtually and in person in Valencia, Spain to […]

Read more →