What is the best practice for Nomad client-server communication over the internet?

We are running a Nomad server cluster on AWS and plan to potentially have thousands of Nomad clients connecting to it from different Cloud Providers (not AWS/GCP/Azure) and private Data Centers.
Exposing servers directly to the public internet is never a good practice so we tried to put the cluster (specifically port 4647) behind a Network Load Balancer and pass the NLB URL to the clients under servers configuration option
i.e -
servers = [ "some.nlb.acme.com" ]
This NLB URL is also the rpc advertise address of the Nomad servers.
With this configuration, it seems like the Nomad client tries to get all 3 expected servers IP from the same URL and ends up having the same IP 3 times -

root@nomad-client:~# nomad node config -servers

(This IP is the one of the NLB interfaces IP in AWS and not a Nomad server IP)

This seems like a bad way to achieve a secured and working setup.
Is there another method / case study / recommendation for setting up a Nomad cluster in a secure way when the clients are not in the same VPC / cloud provider?

1 Like