Nomad datacenter vs Consul datacenter

Nomad
1

Hi, I’m trying to understand how to design and manage the Nomad Multi-Cluster with Consul.

Both Consul and Nomad have a concept of clustering which is called DATACENTER and REGION.

Consul cluster must have a cluster each datacenter.

Nomad cluster must have a cluster each region.

Then I’m considering to create an infrastructure with mesh gateway on Nomad using this job example.

It seems like there are two options to accomplish this scenario.

  1. Create two Consul cluster and create one Nomad cluster with two datacenter.
  2. Create one Consul cluster and create one Nomad cluster with two datacenter.

Which one is appropriate approach? And is it possible to construct mesh gateways on single Consul cluster?

1 Like
2

Hi @morinaga :wave:

As you’ve noted, both Nomad and Consul have a concept of datacenter, but unfortunately they actually represent two different things.

In Nomad, a datacenter is similar to a meta attribute of a client. Since it’s a required job attribute, it can be helpful in grouping a set of clients together that are responsible for running some kind of related jobs.

The actual value of a datacenter doesn’t have any particular meaning to Nomad, other than what you attribute to it. For example, you could split your clients based on your company’s internal departments (research, finance, hr), or by environment (alpha, staging, production). These would be different datacenters and only jobs that have the proper datacenters value would run there.

A region in Nomad is what’s similar to Consul’s datacenter. It defines a low-latency network area, so in a cloud environment this would be like a region in AWS or GCP, or a physical datacenter building with a bunch machines in a on-prem setting.

Now, back to your question :slightly_smiling_face:

Mesh gateways are a way to connect two meshes together that are running in different Consul datacenters (checkout Consul’s doc for more information: Connect Datacenters - Mesh Gateways | Consul by HashiCorp).

With these in mind, the application of mesh gateways depends more on your Consul architecture than Nomad’s. The key thing that you will need is one Consul cluster with two datacenters joined by WAN federation.

For Nomad, it will depend on your actual network topology. Since Consul datacenters are not assumed to have full connectivity between all nodes, it’s probably good to have one Nomad region per Consul datacenter. You can then join multiple regions into a single Nomad cluster with region federation.

Since each end of the mesh gateway will be in a different Nomad region, the Nomad datacenter value is less important.

Sorry for the long answer, but I hope it answers your question.

Let me know if there is anything that is still not clear.

2 Likes
3

@lgfa29 Thank you for your reply. Your explanation is easily understand for me and answered my question correctly.

My understanding is:

  • Nomad and Consul have a concept called “datacenter”, however both aren’t same concept.
  • Let’s say “datacenter” is just a metadata for jobs in Nomad.
  • In Consul, “datacenter” is an actual environment such as AWS region, GCP region, some on-prem site and so on.

So if I understand correctly, above mesh-gateway example should include region explicitly. Because it might cause confusion.

Then now I am really interested about Consul cluster. (Should I create an another topic? That’s would be great I’d be happy to create another one!)

What is the concept or definition of CLUSTER in Consul.

For example, there are two environment(servers and clients are all about Consul):

  1. Three servers these are already joined raft group as a member and two client on the AWS as datacenter “aws” and as domain “consul”.
  2. Three servers these are already joined raft group as a member and two client on the GCP as datacenter “gcp” and as domain “consul”.

These aren’t joined by WAN federation yet at this moment. So can I say there are two cluster?
After are these environment joined by WAN federation to become one cluster?

I’m trying figure out both how should I design cluster and datacenter based on Consul and should I adjust the name of Nomad’s region and Consul’s datacenter.

Answer those question would be great! Thank you.

4

Oof, sorry I missed your reply here…

I guess we could add region to the jobs, though it’s not strictly necessary for using mesh gateways. I think the original idea of that docs page was to provide a simple example that can be easily reproduced? Creating multiple regions in Nomad is a bit more complex.

I’m not a Consul expert, so you will get a better answer from the Consul forum.

From my understanding, there isn’t a specific definition of cluster in Consul. The domain boundaries are set by datacenters and you can use federation to connect multiple datacenters together.

In your example, you have 2 Consul datacenters. You can join then using WAN federation, but they will keep operating independently. Federating the datacenters allows you to automatically forward request between them and (I think) some information can be replicated, such as ACL tokens and intentions.

But as I mentioned, the Consul forum can provide you with a better answer :sweat_smile:

1 Like
5

@lgfa29 I appreciate shared your idea, especially regarding to there isn’t specific definition of cluster in Consul. I might have to post the additional question in Consul forum if I need more investigation about Consul cluster.
I thought I need to aware the deferences between Consul and Nomad in terms of cluster.

Thanks a lot!