How is the Nomad HTTP API served?

Nomad
1

Apologies if this has been explained somewhere else in the docs or the forum but wasn’t able to find it after a while searching.

I would like to understand how the Nomad HTTP API is served. Does a single one of the quorum servers handle that responsibility or are the incoming requests load-balanced among all the servers in the quorum? If the latter, could you give some details on how the load-balancing works?

Thanks a lot and kudos to the Hashicorp team for this amazing product!

1 Like
2

Hi @rostow! That’s a great question with a complicated answer :slightly_smiling_face:

It depends on

  • The nature of the request (e.g. whether “stale” responses are acceptable)
  • Who initially received the request (e.g. Client / Server / Leader)
  • Who needs to answer the request (e.g. a specific Client / Server / Leader)

Nomad implements a lot of plumbing under the hood to forward requests (as RPCs) to the right place if necessary - so ultimately it doesn’t matter which Nomad agent you the API user sends a request to. To elaborate, some API endpoints are answered by specific Client agents (e.g. /v1/node/status), some endpoints can be answered by any of the Server agents in the quorum (e.g. some of the operator commands), and many of the endpoints must be answered by the current server Leader (e.g. job submit, etc.). So there isn’t any explicit “load-balancing” exactly, but rather requests are forwarded to wherever they need to go.

3

Thank you for the detailed answer, that’s a very interesting approach and makes a lot of sense indeed!

I understand now that the requests are forwarded to “where they need to go” / where the information is. Just to confirm though, can an http request land on any Nomad agent (server or client) and still be served in exactly the same way or do the http requests need to go explicitly to the leader?

For some context, we are now working on a use case were we could have thousands of Nodes at peak load, running around 200-300k containers concurrently across the fleet. We need to get large amounts of data out of the system very frequently, from node and job data, to specific allocation task states and resource utilization metrics. Would you recommend for this use case to add custom load balancing for the incoming http requests before they make it to Nomad, and spread them over a few agents?

Wondering if this could be a good use case to add additional non-voting servers to our 5-server quorum, placing them behind an LB and making them solely responsible to serve the API, in order to ensure the server quorum doesn’t have http request handling-related overhead.

4

Yep, a request can land on any Nomad agent and will be automatically forwarded to the correct destination.

Indeed if you’re making a large amount of read requests, adding non-voter servers can help keep load off of the busy quorum servers - Autopilot | Nomad | HashiCorp Developer

Some of the data you’re looking for may also be exposed as metrics - which can be efficiently collected through tools like Prometheus, Datadog, etc. Might be worth making sure you’re taking advantage of those if possible.

5

Awesome, this information is very helpful. Thanks a lot!