Fine Grained Rate Limits for Spinnaker Clouddriver

Feb 7, 2017 by Isaac Mosquera

When using Spinnaker, it queries your cloud provider (AWS, GCP, Azure, Kubernetes, etc) frequently to understand about the state of your existing infrastructure and current deployments. In doing so however, you might run into rate limits imposed by the cloud provider. On AWS you might see an exception similar to the following:

com.amazonaws.AmazonServiceException: CloudWatchAlarm Rate exceeded (Service: AmazonAutoScaling; Status Code: 400; Error Code: Throttling; Request ID: a217a0a2-da7e-11e5-734a-a1917861e2d6)
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1160)
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:748)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:467)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:302)
at com.amazonaws.services.autoscaling.AmazonAutoScalingClient.invoke(AmazonAutoScalingClient.java:3246)

This typically means you have hit the rate limits for the cloud provider.

Recently, the Spinnaker community has added service limits as part of Clouddriver to address growing concerns about these limits. Below is an example configuration for global rate limits for all services that you would place in clouddriver-local.yml.

serviceLimits:
  defaults:
    rateLimit: 20

If you have multiple cloud providers, you can limit each one differently:

serviceLimits:
  cloudProviderOverrides:
    aws:
      rateLimit: 15

You can provide account specific overrides as well in case you have significantly more resources in one account while others have less

serviceLimits:
  accountOverrides:
    test:
      rateLimit: 5
    prod:
      rateLimit: 100

And lastly, you can have more fine-grained control for particular AWS endpoints that might have a different rate limits:

serviceLimits:
  implementationLimits:
    AmazonEC2:
      defaults:
        rateLimit: 200
      accountOverrides:
        prod:
          rateLimit: 500
    AmazonElasticLoadBalancing:
      defaults:
        rateLimit: 10

Using these rate limits will help you avoid hitting the rate limits and potentially make Spinnaker more responsive as the cloud provider clients won’t have to implement back-off strategy to continue to query the infrastructure.

To see more details and other implementationLimits, visit https://docs.armory.io/docs/armory-admin/rate-limit/#fine-grained-rate-limits

As always, we want to provide ridiculously responsive support, let us know how we can help.

Learn More

Share this post:

Recently Published Posts

Navigating AWS Deployment Targets with Armory

Jan 20, 2023

Many organizations look to Amazon Web Services (AWS) to host and deploy their applications in the cloud. However, they’re finding that their deployment tooling, often built as an extension of their legacy continuous integration (CI), is one of the main impediments to adopting cloud services.  Custom-scripted production pipelines built with in-house tooling need to be […]

Read more

Release Roundup – January 2023

Jan 11, 2023

Get the latest product news on Continuous Deployment-as-a-Service and the most recent release for Continuous Deployment Self Hosted, 2.28.2. Welcome to 2023!  Just like every organization, Armory is looking for ways to improve our practices and deliver more value (and faster!) to you, our customers. That’s why our engineering team is working to deliver features, […]

Read more

Learn Continuous Deployment with Armory and Wilco

Jan 6, 2023

Armory is excited to announce we have launched an interactive, narrative-driven developer experience that teaches continuous deployment concepts. And now you can try it out for yourself! Wilco, also known as the “flight simulator” for software developers, allows companies to create engaging interactive developer challenges (called quests) that enable developers to acquire and practice skills […]

Read more