As you’ve noted, both Nomad and Consul have a concept of datacenter, but unfortunately they actually represent two different things.
In Nomad, a datacenter is similar to a meta attribute of a client. Since it’s a required job attribute, it can be helpful in grouping a set of clients together that are responsible for running some kind of related jobs.
The actual value of a datacenter doesn’t have any particular meaning to Nomad, other than what you attribute to it. For example, you could split your clients based on your company’s internal departments (
hr), or by environment (
production). These would be different datacenters and only jobs that have the proper
datacenters value would run there.
A region in Nomad is what’s similar to Consul’s datacenter. It defines a low-latency network area, so in a cloud environment this would be like a region in AWS or GCP, or a physical datacenter building with a bunch machines in a on-prem setting.
Now, back to your question
Mesh gateways are a way to connect two meshes together that are running in different Consul datacenters (checkout Consul’s doc for more information: Connect Datacenters - Mesh Gateways | Consul by HashiCorp).
With these in mind, the application of mesh gateways depends more on your Consul architecture than Nomad’s. The key thing that you will need is one Consul cluster with two datacenters joined by WAN federation.
For Nomad, it will depend on your actual network topology. Since Consul datacenters are not assumed to have full connectivity between all nodes, it’s probably good to have one Nomad region per Consul datacenter. You can then join multiple regions into a single Nomad cluster with region federation.
Since each end of the mesh gateway will be in a different Nomad region, the Nomad datacenter value is less important.
Sorry for the long answer, but I hope it answers your question.
Let me know if there is anything that is still not clear.