Best practices for using the Nomad Autoscaler with SaaS APM Tools?

Nomad
1

I have been gradually updating various services to use the autoscaler over the past month or so and am starting to encounter issues with scaling off of Datadog metrics. The main issue is that Datadog, as well as other SaaS APM tools, have fairly aggressive rate limits for querying timeseries data using their APIs. For Datadog it is possible to workaround these limits by raising them, but this doesn’t seem scalable as they will need to be perpetually raised as more scaling checks and services are created.

I was wondering if anyone using the autoscaler has any advice or best practices for limiting the amount of queries the autoscaler makes against APM tools such as aggregating queries across multiple services?

2 Likes
2

Hi @tyler-domitrovich and thanks for the interesting question. The current Datadog APM plugin does not have the ability to perform aggregated queries and at a more core level, the Autoscaler APM plugin interface does not accommodate this either. I will raise this idea against the Nomad Autoscaler repository in order to start some initial discussions.

In the meantime, a workaround could be to modify the default_evaluation_interval to a less aggressive value to reduce the overall number of API calls made.

jrasell and the Nomad team.

1 Like