Does that mean that a Nomad server can talk to a Consul server?
Or maybe even that Nomad servers are not supposed to talk to any Consuls?
The more I think about it, the more confused I get.
Here’s what I’m thinking of doing:
6 nodes
lets say first 3 are “alpha”, the other 3 are “beta” nodes
alpha nodes will run Nomad servers and Consul servers
beta nodes will run Nomad agents and Consul agents
Nomads’ config on alpha will point to Consul on alpha (server talks to server).
Nomads’ config on beta will point to Consul on beta (agent talks to agent).
An important requirement is that each Nomad agent talks to a unique Consul agent. […]
It is about that you should run unique Consul on the same machine as Nomad for every Nomad. That consul can be a server.
Does that mean that a Nomad server can talk to a Consul server?
Nomad server can talk to a Consul server.
maybe even that Nomad servers are not supposed to talk to any Consuls?
No. Nomad servers can registers themselves in consul.
Am I doing this right?
If you have 6 nodes, then you have 6 nomads and 6 consuls, one per each machine, and every nomad is configured to talk to his own unique consul (simply on 127.0.0.1).
Now, some of these nomads and consuls can be servers, in any combination. What is important is that you don’t mix - no two nomads talk to the same consul.
Running a nomad agent and consul server on the same machine is similar (bad) idea like running nomad agents+server together. Nomad client may happen to schedule big work on the machine and because of that other services may have problems with services and nomad or consul can stall. I actually run nomad client+server and consul server on the same machine, I keep them in a different datacenter in nomad configuration, so I can control the workload nomad puts there.