How does Nomad restrict which clients are allowed to join the cluster?

Nomad
1

Hi there, I am fairly new into Nomad and likely still missing the big picture.

For my test use case I set up a production-grade Nomad cluster with three Hetzner Cloud VMs (one server and two clients). The VMs are conntected using an internal network. The server advertises itself at 10.0.0.0 (but binds to 0.0.0.0) and the clients are using the internal IP address to join the cluster.

At Hetzner every VM always comes with an external IP (which is a good thing because I want to access the dashboard and webservices deployed in Nomad via public Internet).

I’m now wondering how Nomad decides which clients are allowed to join the cluster. I tried joining using an external VM via the server’s public IP address and it actually registered with the cluster (even though it got marked as “down”).

Is there any way besides using TLS certificates to ensure that only specific Nomad clients can join a cluster? I’d like not to use the “verify_https_client” option for that because it results in a lot of overhead to distribute these certificates for CLI and Terraform.

Is there any documentation regarding how this works? I’ve read through Nomad architecture and network connectivity, but was not able to find answers to these questions.

Any help is greatly appreciated :slight_smile:

2

It’s been a (long) while since I’ve had to manually enter any IPs, but since I don’t think there’s auto-join support for Hetzner Cloud…

I think I tend to explicitly set the 3 advertised addresses rather than the bind_addr

i.e.

advertise {
  http = "{PRIVATE_IPV4}"
  rpc = "{PRIVATE_IPV4}"
  serf = "{PRIVATE_IPV4}"
}

If you are going to setup a production Nomad cluster, I would strongly advise you to:

  1. use 3 Nomad servers
  2. do not provision servers that have external IPs – us private IPs only and instead use an HTTPS load balancer to load balance to to your Nomad servers on port 4646 instead (and restrict access to your network)
  3. encrypt your Nomad servers’ gossip traffic
  4. use TLS certs
  5. highly recommend setting up a Consul cluster as well

As for how it connects, I believe you just need to have the correct connection info

3

@Neutrollized Thank you so much for your detailed response. So I guess in a way my understanding was right - I have to actually restrict access to the cluster using internal networks and/or TLS to prevent agents from joining. There is no Kubernetes-style join feature which requires the agents to have knowledge of a join-token or similar.

Thanks for confirming :slight_smile: