I want servers to communicate using a different subnet than clients to servers. So I have a different ip address for http and for serf.
advertise {
http = "172.29.192.150:4646"
rpc = "172.29.192.150:4647"
serf = "10.120.18.150:4648"
}
I use DNS from consul, with nomad autoregistration. Sadly, consul returns all ip addresses for nomad, also ip addresses for serf.
$ dig nomad.service.consul....
nomad.service.consul.. 0 IN A 172.29.192.150
nomad.service.consul.. 0 IN A 172.29.192.51
nomad.service.consul.. 0 IN A 10.120.18.150
nomad.service.consul.. 0 IN A 10.120.18.52
nomad.service.consul.. 0 IN A 172.29.248.59
nomad.service.consul.. 0 IN A 10.120.18.153
Nomad clients however do not have access to the seft network. Specifying NOMAD_ADDR=http://nomad.service.consul
occasionally results in “not found” errors when clients try to connect with 10.120 network instead of 172.29. Not only that, when using consul for discovery of Nomad servers, Nomad clients specifically try to connect to Seft port. According to Agent Configuration | Nomad | HashiCorp Developer, Seft port must be reachable for Nomad servers, and Nomad clients should use RPC. Is the documentation mixing Serf and RPC ports?
How should I specify NOMAD_ADDR
for CLI tool so that they will reach out to http ip of nomad servers? Can I configure Nomad servers so that they communicate together using different ip thatn comminucate from CLI and with Nomad clients?