[Nomad][autoscaler] Scaling down gives 429 error

Nomad
1

When the autoscaler is evaluating a policy it is giving 429 error which causes problem in scale in/scale out. Getting error as

  | ), failed to drain node: Unexpected response code: 429 (Your IP is issuing too many concurrent connections, please rate limit your calls
  | ), context done while monitoring node drain: received error while draining node: Error monitoring node: Unexpected response code: 429 (Your IP is issuing too many concurrent connections, please rate limit your calls

Do I need to change any setting on nomad client/server? Attaching the logs for reference.
autoscaler_logs_12thOct2022.txt (249.3 KB)

2

Hi @jdk2588,

That error is coming from the Nomad agent that the autoscaler is connected to and means there are too many concurrent connections. How many policies do you currently have configured that are held within Nomad, at what interval are they evaluated, and do you have other applications that are making API calls to the same agent?

Thanks,
jrasell and the Nomad team

3

I have the same issue with nomad autoscaler

2022-12-12T10:55:16.272Z [DEBUG] internal_plugin.nomad-target: failed to read job scale status, retrying: Job=XXX
  error=
  | Unexpected response code: 429 (Your IP is issuing too many concurrent connections, please rate limit your calls
  | )

I have 1 instance of auto-scaler process and around 200 jobs with scaling enabled but with "Policy": null - I want to have manual scaling without automatic.

Do you have any advice how to handle this case?

4

In my case increasing http_max_conns_per_client limit to 500 on each nomad instance helped