Skip to main content
← Back to posts

Fine Grained Rate Limits for Spinnaker Clouddriver

Feb 7, 2017 by Isaac Mosquera

When using Spinnaker, it queries your cloud provider (AWS, GCP, Azure, Kubernetes, etc) frequently to understand about the state of your existing infrastructure and current deployments. In doing so however, you might run into rate limits imposed by the cloud provider. On AWS you might see an exception similar to the following:

com.amazonaws.AmazonServiceException: CloudWatchAlarm Rate exceeded (Service: AmazonAutoScaling; Status Code: 400; Error Code: Throttling; Request ID: a217a0a2-da7e-11e5-734a-a1917861e2d6)
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1160)
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:748)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:467)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:302)
at com.amazonaws.services.autoscaling.AmazonAutoScalingClient.invoke(AmazonAutoScalingClient.java:3246)

This typically means you have hit the rate limits for the cloud provider.

Recently, the Spinnaker community has added service limits as part of Clouddriver to address growing concerns about these limits. Below is an example configuration for global rate limits for all services that you would place in clouddriver-local.yml.

serviceLimits:
  defaults:
    rateLimit: 20

If you have multiple cloud providers, you can limit each one differently:

serviceLimits:
  cloudProviderOverrides:
    aws:
      rateLimit: 15

You can provide account specific overrides as well in case you have significantly more resources in one account while others have less

serviceLimits:
  accountOverrides:
    test:
      rateLimit: 5
    prod:
      rateLimit: 100

And lastly, you can have more fine-grained control for particular AWS endpoints that might have a different rate limits:

serviceLimits:
  implementationLimits:
    AmazonEC2:
      defaults:
        rateLimit: 200
      accountOverrides:
        prod:
          rateLimit: 500
    AmazonElasticLoadBalancing:
      defaults:
        rateLimit: 10

Using these rate limits will help you avoid hitting the rate limits and potentially make Spinnaker more responsive as the cloud provider clients won’t have to implement back-off strategy to continue to query the infrastructure.

To see more details and other implementationLimits, visit https://docs.armory.io/docs/armory-admin/rate-limit/#fine-grained-rate-limits

As always, we want to provide ridiculously responsive support, let us know how we can help.

Learn More

Recently Published Posts

July 26, 2021
|
by Phebe Vickers

A day in the life of a TAM

I’ve been asked what a Technical Account Manager (TAM) does so I wanted to take the opportunity to illustrate it by walking through a standard day in the life. Before we can look at what a day in a life of a TAM is, I should provide some background in what is a TAM and […]

Read more →

June 29, 2021
|
by Nikema Prophet

Nikema’s Spinnaker Summit 2021 Recap

My Second Spinnaker Summit is in the Books! Last week I attended and spoke at my second Spinnaker Summit. Like last year’s summit, it was fully virtual. This time Spinnaker Summit was co-located with cdCon and took place on the Hopin platform. Last year, I spoke on a panel about Black professionals a few months […]

Read more →

June 28, 2021
|
by Stephen Atwell

Announcing General Availability of Armory Policy Engine Plugin

Armory Policy Engine provides support for automating policy compliance with Spinnaker. Policy Engine Plugin is the latest version of Policy Engine and adds support for both advanced role-based access control (RBAC) use-cases and open source Spinnaker. The release of Policy Engine Plugin comes with new documentation, including a library of example policies from across Armory’s […]

Read more →