Skip to main content
← Back to posts

Fine Grained Rate Limits for Spinnaker Clouddriver

Feb 7, 2017 by Isaac Mosquera

When using Spinnaker, it queries your cloud provider (AWS, GCP, Azure, Kubernetes, etc) frequently to understand about the state of your existing infrastructure and current deployments. In doing so however, you might run into rate limits imposed by the cloud provider. On AWS you might see an exception similar to the following:

com.amazonaws.AmazonServiceException: CloudWatchAlarm Rate exceeded (Service: AmazonAutoScaling; Status Code: 400; Error Code: Throttling; Request ID: a217a0a2-da7e-11e5-734a-a1917861e2d6)
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1160)
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:748)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:467)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:302)
at com.amazonaws.services.autoscaling.AmazonAutoScalingClient.invoke(AmazonAutoScalingClient.java:3246)

This typically means you have hit the rate limits for the cloud provider.

Recently, the Spinnaker community has added service limits as part of Clouddriver to address growing concerns about these limits. Below is an example configuration for global rate limits for all services that you would place in clouddriver-local.yml.

serviceLimits:
  defaults:
    rateLimit: 20

If you have multiple cloud providers, you can limit each one differently:

serviceLimits:
  cloudProviderOverrides:
    aws:
      rateLimit: 15

You can provide account specific overrides as well in case you have significantly more resources in one account while others have less

serviceLimits:
  accountOverrides:
    test:
      rateLimit: 5
    prod:
      rateLimit: 100

And lastly, you can have more fine-grained control for particular AWS endpoints that might have a different rate limits:

serviceLimits:
  implementationLimits:
    AmazonEC2:
      defaults:
        rateLimit: 200
      accountOverrides:
        prod:
          rateLimit: 500
    AmazonElasticLoadBalancing:
      defaults:
        rateLimit: 10

Using these rate limits will help you avoid hitting the rate limits and potentially make Spinnaker more responsive as the cloud provider clients won’t have to implement back-off strategy to continue to query the infrastructure.

To see more details and other implementationLimits, visit https://docs.armory.io/docs/armory-admin/rate-limit/#fine-grained-rate-limits

As always, we want to provide ridiculously responsive support, let us know how we can help.

Learn More

Recently Published Posts

October 26, 2021
|
by Stephen Atwell

How to Take the Pain of Rollbacks out of Deployments

  Software applications have become an integral part of the business climate in most modern organizations. With an ever-increasing demand for new features and enhancement of already-existing ones, software teams constantly face novel challenges, and the pace of software development is growing by the day. To keep up with this fast-paced business climate, software teams […]

Read more →

October 20, 2021
|
by Jason McIntosh

Monitoring Spinnaker: Part 1

Overview One of the questions that comes up a lot is how you monitor Spinnaker itself.  Not the apps Spinnaker is deploying, but Spinnaker itself and how it’s performing.  This is a question that has a lot of different answers. There are a few guidelines, but many of the answers are the same as how […]

Read more →

October 18, 2021
|
by David Morgenthaler

The Importance of Patents: Interview with Nick Petrella, Head of Legal

    In honor of Armory’s recent acquisition of a patent for continuous software deployment, we sat down with Nick Petrella, Head of Legal, for a casual conversation covering a wide range of subjects, from patent law to Nick’s background as a software engineer and why he made the leap to the law. Check out […]

Read more →