Production architecture for consul and nomad deployment

Consul
1

This probably is a simple question and I would imagine this has been discussed somewhere but I honestly can’t find the answer.
Consul: https://www.consul.io/docs/internals/architecture.html
Nomad: https://nomadproject.io/docs/internals/architecture/

In a single production data center what is the recommended layout for consul and nomad together?
I assume for max stability 5 nodes is recommended.

However should each consul and nomad instance be on the same VM or the same hardware or same rack?

In a 5 node cluster it might not matter however does it matter more in a 3 node cluster?

For instance:

Machine1: Nomad/Consul
Machine2: Nomad/Consul
Machine3: Nomad/Consul

vs

Machine1c: Consul
Machine2c: Consul
Machine3c: Consul
Machine1n: Nomad
Machine2n: Nomad
Machine3n: Nomad

vs

Machine1: Nomad/Consul
Machine2: Nomad/Consul
Machine3: Nomad/Consul
Machine4: Nomad/Consul
Machine5: Nomad/Consul

vs

Machine1c: Consul
Machine2c: Consul
Machine3c: Consul
Machine4c: Consul
Machine5c: Consul
Machine1n: Nomad
Machine2n: Nomad
Machine3n: Nomad
Machine4n: Nomad
Machine5n: Nomad

2

I would prefer 3 Consul nodes and 3 Nomad nodes as clusters. Every nodes does one thing and one thing only, so your second option. If you need more, you can add some.

I don’t think that it’s still a “bare metal is better than VMs”-thing. Not even with databases, so vms should be fine.

3

Have you seen this Nomad Reference Architecture page?

Deployment Topology within a Single Region

…

Each cluster is expected to have either three or five servers. This strikes a balance between availability in the case of failure and performance, as Raft consensus gets progressively slower as more servers are added.

…

Nomad Cluster
Nomad Cluster855×522 35.7 KB

Basically what @Wolfsrudel is saying. You have a 3-5 node cluster of Consul servers, a 3-5 node cluster of Nomad servers with Consul clients, and then your target machines running Nomad and Consul clients.

4

Thanks.
Just to be clear VMs are fine however they should be on different machines for stability?
1 VM per machine so 3 VMs for consul, 3 for nomad which gives 6 VMs and 6 physical servers.

5

I don’t think that you need that. You can have ha and performance scaling at the virtualization layer and let it handle that. Everything >= 2 is a cluster and you can start with that. In the worst case one bare metal server should be able to take over all vms. Just in case of a disaster.

What you describe was a setup in former times. One bare metal server for one thing. :wink:

6

Yes your absolutely true.
Wasn’t thinking of a VM high availability system.

With that each VM/container could effectively be its own server.
An amazing concept really.