Dockerized Consul & Nomad Cluster difficulties

Hello Hashi Community,

I have recently begun mapping out a single-VM deployment of our in-house application which has been running as distributed microservices on separate VMs using Consul, Nomad and Vault to manage KV, scheduling and secrets, respectively.

As a POC I’m attempting to set up the Consul and Nomad framework using docker images of each to facilitate multi-server quorum on the same machine in lieu of having multiple VMs to work with.

I was able to create the Consul cluster without issues using Docker Compose and the default docker bridge network.

The difficulty I’m having is that I can’t seem to get Nomad to cluster using Consul (auto join) using the same bridge network and feel that I’ve exhausted all advertise and bind options (starting with the defaults.

Fingerprinting seems to be working, and Nomad will register with Consul but will not pass its health check and ultimately cannot elect a leader.


The configuration I’ve been trying is 3 Consul server, 1 Consul client x 3 Nomad Servers which are also clients.

Here are my Nomad configs -

image

Docker Compose (just the Nomad section): please let me know if I can supply any other configuration info or details. Thank you all in advance for any input!

nomad-server-1:
    image: vptech/nomad:1.0.4
    container_name: nomad-server1
    command: /bin/nomad agent -config=/server1.hcl -config=/client.hcl -config=/base.hcl
    environment:
      NOMAD_RUN_ROOT: 1
    ports:
      - 4646:4646
      - 4647:4647
      - 4648:4648

    restart: always
    privileged: true
    cap_add: 
    - SYS_ADMIN
    - NET_ADMIN
    - chown
    - dac_override
    - fsetid
    - fowner
    - mknod
    - net_raw
    - setgid
    - setuid
    - setfcap
    - setpcap
    - net_bind_service
    - sys_chroot
    - kill
    - audit_write
    - IPC_LOCK

    volumes:
    - /var/run/docker.sock:/var/run/docker.sock
    - ./nomad/base.hcl:/base.hcl
    - ./nomad/client.hcl:/client.hcl
    - ./nomad/server1.hcl:/server1.hcl
    - /tmp:/tmp

  nomad-server-2:
    image: vptech/nomad:1.0.4
    container_name: "nomad-server2"
    command: /bin/nomad agent -config=/server2.hcl -config=/client.hcl -config=/base.hcl
    environment:
      NOMAD_RUN_ROOT: 1 
    ports:
      - 5646:5646
      - 5647:5647
      - 5648:5648

    restart: always
    privileged: true
    cap_add: 
    - SYS_ADMIN
    - NET_ADMIN
    - chown
    - dac_override
    - fsetid
    - fowner
    - mknod
    - net_raw
    - setgid
    - setuid
    - setfcap
    - setpcap
    - net_bind_service
    - sys_chroot
    - kill
    - audit_write
    - IPC_LOCK

    volumes:
    - /var/run/docker.sock:/var/run/docker.sock
    - ./nomad/base.hcl:/base.hcl
    - ./nomad/client.hcl:/client.hcl
    - ./nomad/server2.hcl:/server2.hcl
    - /tmp:/tmp

  nomad-server-3:
    image: vptech/nomad:1.0.4
    container_name: "nomad-server3"
    command: /bin/nomad agent -config=/server3.hcl -config=/client.hcl -config=/base.hcl
    environment:
      NOMAD_RUN_ROOT: 1 
    ports:
      - 6646:6646
      - 6647:6647
      - 6648:6648
    restart: always
    privileged: true

    cap_add: 
    - SYS_ADMIN
    - NET_ADMIN
    - chown
    - dac_override
    - fsetid
    - fowner
    - mknod
    - net_raw
    - setgid
    - setuid
    - setfcap
    - setpcap
    - net_bind_service
    - sys_chroot
    - kill
    - audit_write
    - IPC_LOCK

    volumes:
    - /var/run/docker.sock:/var/run/docker.sock
    - ./nomad/base.hcl:/base.hcl
    - ./nomad/client.hcl:/client.hcl
    - ./nomad/server3.hcl:/server3.hcl
    - /tmp:/tmp