Nomad + Consul Connect

Nomad
1

Summary: Cannot get Nomad to work with Consul Connect sidecar.

The setup is this: A cluster of three VirtualBox VMs running Ubuntu 16.04.6 with Docker, Consul 1.6.1 and Nomad 0.10.0-connect1, spun up by Vagrant on MacOS. If you want to try it out, you can use my images and follow my setup.

What works: running a simple docker container on all host (see trivial.nomad).

What does now work: Running countdash.nomad taken from the official example. According to the Consul UI, the “count-dashboard-sidecar-proxy” and " count-api-sidecar-proxy" services both display an error on “Connect Sidecars Listening” with this error on stdout:

dial tcp 127.0.0.1:25420: connect: connection refused

From journalctl, I get these errors on the host:

manager-3 nomad[2279]: {
manager-3 nomad[2279]:     "cniVersion": "0.4.0",
manager-3 nomad[2279]:     "interfaces": [
manager-3 nomad[2279]:         {
manager-3 nomad[2279]:             "name": "nomad",
manager-3 nomad[2279]:             "mac": "36:53:73:a6:fd:0a"
manager-3 nomad[2279]:         },
manager-3 nomad[2279]:         {
manager-3 nomad[2279]:             "name": "veth9e6a6671",
manager-3 nomad[2279]:             "mac": "f2:89:40:b3:c3:e9"
manager-3 nomad[2279]:         },
manager-3 nomad[2279]:         {
manager-3 nomad[2279]:             "name": "eth0",
manager-3 nomad[2279]:             "mac": "42:4d:51:91:11:83",
manager-3 nomad[2279]:             "sandbox": "/var/run/docker/netns/b8a05ee35ccc"
manager-3 nomad[2279]:         }
manager-3 nomad[2279]:     ],
manager-3 nomad[2279]:     "ips": [
manager-3 nomad[2279]:         {
manager-3 nomad[2279]:             "version": "4",
manager-3 nomad[2279]:             "interface": 2,
manager-3 nomad[2279]:             "address": "172.26.66.2/23",
manager-3 nomad[2279]:             "gateway": "172.26.66.1"
manager-3 nomad[2279]:         }
manager-3 nomad[2279]:     ],
manager-3 nomad[2279]:     "routes": [
manager-3 nomad[2279]:         {
manager-3 nomad[2279]:             "dst": "0.0.0.0/0",
manager-3 nomad[2279]:             "gw": "172.26.66.1"
manager-3 nomad[2279]:         }
manager-3 nomad[2279]:     ],
manager-3 nomad[2279]:     "dns": {}
manager-3 nomad[2279]: }    2019-09-14T14:58:16.323Z [WARN ] client.alloc_runner.runner_hook.consul_socket: unable to create unix sockets for Consul; can not determine Consul gRPC endpoint: alloc_id=3eb1d8d3-fe87-dc60-aca7-f7ce10616a91 error="parse 10.1.2.13:8500: first path segment in URL cannot contain colon" consul_addr=10.1.2.13:8500
manager-3 consul[2054]:     2019/09/14 14:58:16 [ERR] roots watch error: invalid type for roots response: <nil>
manager-3 consul[2054]:     2019/09/14 14:58:16 [ERR] leaf watch error: invalid type for leaf response: <nil>
manager-3 consul[2054]:     2019/09/14 14:58:16 [INFO] agent: Synced service "_nomad-task-3eb1d8d3-fe87-dc60-aca7-f7ce10616a91-group-api-count-api-9001"
manager-3 consul[2054]: roots watch error: invalid type for roots response: <nil>
manager-3 consul[2054]: leaf watch error: invalid type for leaf response: <nil>
manager-3 consul[2054]: agent: Synced service "_nomad-task-3eb1d8d3-fe87-dc60-aca7-f7ce10616a91-group-api-count-api-9001"

Any clues to what is wrong would be highly appreciated!

2

It turns out I was missing three things: Use Nomad 0.10.0-beta1 (instead of -connect1) and have this in /etc/consul.d/consul.hcl:

connect {
  enabled = true
}
ports {
  grpc = 8502
}

The repo is updated with these changes, for anyone interested.

2 Likes
3

Hi @jpsecher, minor nitpick here; the connect stanza is needed only on the server(s).

I believe that the grpc section is only needed on the non-servers, though not sure.

Ref from Gitter room:
https://gitter.im/hashicorp-nomad/Lobby?at=5d78c4fda08e2b4bd2c6ba5f