Do you have to use Connect to have cross service communcation?

So I have to client nodes one on docker for desktop and one on local machine. The server sits in the docker for desktop kubernetes cluster. I’m trying to link up the services in k8s with the services on my local. Can I just use the [service-name].consul dns name or do I have to have everything proxied through connect… I need all services to be connected to all other services…

Hi Callum,
First of all I’d want to make sure that the client node on your local machine can talk to the client in docker for desktop and the server in docker for desktop and vice versa. You’ll need to make sure your advertise-ips are set properly. You can check that it’s working if consul members returns the same response on all clients and server and if there are no errors in the logs.

I think for this to work, you’ll need to find a Docker for desktop networking configuration that supports the same IP address being routable from inside Docker and from your host. For example, the Consul client in Docker has one IP address. This needs to be routable from the Consul server running inside Docker and from the Consul client running on your host. I’m honestly not sure if thats possible.

Once you’ve got your cluster configured properly we can address the routing. Again the problem is routable IPs. if the cluster is working then presumably your services can have IPs that are equally routable from host and docker. If this is the case, then you can use Consul DNS. The lookup [service-name].consul will return an IP that will work from inside Docker and outside.

Hi Luke,

Thank you for your reply, really valuable information.

So currently when I run consul members it does work as expected, as in I can see all the docker in desktop node and the server in k8s.

Yeah, I assumed because docker for mac bind directly to your loop back, that this would be very difficult…

I wonder if I ran a docker-machine which does get an IP, then hook that up to the local k8s cluster

Here are some logs from local consul

2020/01/22 11:35:19 [DEBUG] memberlist: Failed ping: docker-for-desktop (timeout reached)
2020/01/22 11:35:21 [WARN] agent: Check "service:graph-api-local-sidecar-proxy:1" socket connection failed: dial tcp 127.0.0.1:21000: connect: connection refused
2020/01/22 11:35:21 [WARN] manager: No servers available
2020/01/22 11:35:25 [DEBUG] memberlist: Stream connection from=192.168.10.222:49888
2020/01/22 11:35:25 [INFO] serf: EventMemberJoin: wet-fish-consul-server-0 10.1.0.129
2020/01/22 11:35:25 [INFO] consul: adding server wet-fish-consul-server-0 (Addr: tcp/10.1.0.129:8300) (DC: k8s)
2020/01/22 11:35:25 [DEBUG] tlsutil: OutgoingRPCWrapper with version 1
2020/01/22 11:35:27 [INFO] memberlist: Suspect docker-for-desktop has failed, no acks received
2020/01/22 11:35:28 [ERR] memberlist: Push/Pull with docker-for-desktop failed: dial tcp 10.1.0.128:8301: i/o timeout
2020/01/22 11:35:28 [DEBUG] memberlist: Failed ping: wet-fish-consul-server-0 (timeout reached)
2020/01/22 11:35:31 [WARN] agent: Check "service:graph-api-local-sidecar-proxy:1" socket connection failed: dial tcp 127.0.0.1:21000: connect: connection refused
2020/01/22 11:35:31 [INFO] memberlist: Marking docker-for-desktop as failed, suspect timeout reached (0 peer confirmations)
2020/01/22 11:35:31 [INFO] serf: EventMemberFailed: docker-for-desktop 10.1.0.128
2020/01/22 11:35:35 [ERR] consul: "ConnectCA.Roots" RPC failed to server 10.1.0.129:8300: rpc error getting client: failed to get conn: dial tcp <nil>->10.1.0.129:8300: i/o timeout
2020/01/22 11:35:35 [ERR] consul: "Coordinate.Update" RPC failed to server 10.1.0.129:8300: rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection
2020/01/22 11:35:35 [ERR] consul: "Catalog.NodeServices" RPC failed to server 10.1.0.129:8300: rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection
2020/01/22 11:35:35 [ERR] agent: failed to sync remote state: rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection
2020/01/22 11:35:35 [ERR] consul: "Intention.Match" RPC failed to server 10.1.0.129:8300: rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection
2020/01/22 11:35:35 [ERR] agent: Coordinate update error: rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection
2020/01/22 11:35:35 [DEBUG] tlsutil: OutgoingRPCWrapper with version 1
2020/01/22 11:35:36 [INFO] memberlist: Suspect wet-fish-consul-server-0 has failed, no acks received
2020/01/22 11:35:36 [DEBUG] memberlist: Failed ping: wet-fish-consul-server-0 (timeout reached)
2020/01/22 11:35:37 [DEBUG] serf: forgoing reconnect for random throttling
2020/01/22 11:35:40 [INFO] memberlist: Marking wet-fish-consul-server-0 as failed, suspect timeout reached (0 peer confirmations)
2020/01/22 11:35:40 [INFO] serf: EventMemberFailed: wet-fish-consul-server-0 10.1.0.129
2020/01/22 11:35:40 [INFO] consul: removing server wet-fish-consul-server-0 (Addr: tcp/10.1.0.129:8300) (DC: k8s)
2020/01/22 11:35:40 [INFO] serf: EventMemberJoin: wet-fish-consul-server-0 10.1.0.129
2020/01/22 11:35:40 [INFO] consul: adding server wet-fish-consul-server-0 (Addr: tcp/10.1.0.129:8300) (DC: k8s)
2020/01/22 11:35:41 [WARN] agent: Check "service:graph-api-local-sidecar-proxy:1" socket connection failed: dial tcp 127.0.0.1:21000: connect: connection refused
2020/01/22 11:35:44 [INFO] memberlist: Suspect wet-fish-consul-server-0 has failed, no acks received
2020/01/22 11:35:45 [DEBUG] manager: pinging server "wet-fish-consul-server-0 (Addr: tcp/10.1.0.129:8300) (DC: k8s)" failed: rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection
2020/01/22 11:35:45 [DEBUG] manager: No healthy servers during rebalance, aborting
2020/01/22 11:35:45 [ERR] consul: "ConnectCA.Roots" RPC failed to server 10.1.0.129:8300: rpc error getting client: failed to get conn: dial tcp <nil>->10.1.0.129:8300: i/o timeout
2020/01/22 11:35:45 [ERR] consul: "Catalog.NodeServices" RPC failed to server 10.1.0.129:8300: rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection
2020/01/22 11:35:45 [ERR] agent: failed to sync remote state: rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection
2020/01/22 11:35:45 [DEBUG] tlsutil: OutgoingRPCWrapper with version 1
2020/01/22 11:35:45 [DEBUG] memberlist: Failed ping: wet-fish-consul-server-0 (timeout reached)
2020/01/22 11:35:51 [WARN] agent: Check "service:graph-api-local-sidecar-proxy:1" socket connection failed: dial tcp 127.0.0.1:21000: connect: connection refused
2020/01/22 11:35:53 [INFO] memberlist: Suspect wet-fish-consul-server-0 has failed, no acks received
2020/01/22 11:35:53 [DEBUG] memberlist: Failed ping: wet-fish-consul-server-0 (timeout reached)
2020/01/22 11:35:55 [ERR] consul: "ConnectCA.Roots" RPC failed to server 10.1.0.129:8300: rpc error getting client: failed to get conn: dial tcp <nil>->10.1.0.129:8300: i/o timeout
2020/01/22 11:35:55 [ERR] consul: "Intention.Match" RPC failed to server 10.1.0.129:8300: rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection
2020/01/22 11:35:55 [DEBUG] tlsutil: OutgoingRPCWrapper with version 1

Connecting to the web ui of Consul in K8s, it has this Agent alive and reachable to say about the local machine agent.

However, I can’t seem to add a new service definition by adding a new config file to the config directory and doing consul reload. If I do straight kill and respawn this is the output:

==> Log data will now stream in as it occurs:

2020/01/22 11:52:45 [DEBUG] tlsutil: Update with version 1
2020/01/22 11:52:45 [INFO] serf: EventMemberJoin: local-client 192.168.10.222
2020/01/22 11:52:45 [INFO] agent: Started DNS server 127.0.0.1:9600 (tcp)
2020/01/22 11:52:45 [INFO] agent: Started DNS server 127.0.0.1:9600 (udp)
2020/01/22 11:52:45 [WARN] manager: No servers available
2020/01/22 11:52:45 [ERR] roots watch error: error filling agent cache: No known Consul servers
2020/01/22 11:52:45 [WARN] manager: No servers available
2020/01/22 11:52:45 [WARN] manager: No servers available
2020/01/22 11:52:45 [WARN] manager: No servers available
2020/01/22 11:52:45 [ERR] roots watch error: error filling agent cache: No known Consul servers
2020/01/22 11:52:45 [ERR] roots watch error: error filling agent cache: No known Consul servers
2020/01/22 11:52:45 [WARN] manager: No servers available
2020/01/22 11:52:45 [WARN] manager: No servers available
2020/01/22 11:52:45 [ERR] intentions watch error: error filling agent cache: No known Consul servers
2020/01/22 11:52:45 [WARN] manager: No servers available
2020/01/22 11:52:45 [ERR] roots watch error: error filling agent cache: No known Consul servers
2020/01/22 11:52:45 [WARN] manager: No servers available
2020/01/22 11:52:45 [ERR] intentions watch error: error filling agent cache: No known Consul servers
2020/01/22 11:52:45 [ERR] intentions watch error: error filling agent cache: No known Consul servers
2020/01/22 11:52:45 [WARN] manager: No servers available
2020/01/22 11:52:45 [ERR] intentions watch error: error filling agent cache: No known Consul servers
2020/01/22 11:52:45 [WARN] manager: No servers available
2020/01/22 11:52:45 [INFO] agent: Started HTTP server on 127.0.0.1:9500 (tcp)
2020/01/22 11:52:45 [WARN] manager: No servers available
2020/01/22 11:52:45 [ERR] leaf watch error: error filling agent cache: No known Consul servers
2020/01/22 11:52:45 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce k8s mdns os packet scaleway softlayer triton vsphere
2020/01/22 11:52:45 [INFO] agent: Joining LAN cluster...
2020/01/22 11:52:45 [WARN] manager: No servers available
2020/01/22 11:52:45 [INFO] agent: (LAN) joining: [kubernetes.docker.internal]
2020/01/22 11:52:45 [ERR] leaf watch error: error filling agent cache: No known Consul servers
2020/01/22 11:52:45 [WARN] manager: No servers available
2020/01/22 11:52:45 [ERR] leaf watch error: error filling agent cache: No known Consul servers
2020/01/22 11:52:45 [INFO] agent: started state syncer
==> Consul agent running!
2020/01/22 11:52:45 [WARN] manager: No servers available
2020/01/22 11:52:45 [WARN] manager: No servers available
2020/01/22 11:52:45 [ERR] leaf watch error: error filling agent cache: No known Consul servers
2020/01/22 11:52:45 [WARN] manager: No servers available
2020/01/22 11:52:45 [WARN] manager: No servers available
2020/01/22 11:52:45 [WARN] manager: No servers available
2020/01/22 11:52:45 [ERR] agent: failed to sync remote state: No known Consul servers
2020/01/22 11:52:45 [DEBUG] memberlist: Initiating push/pull sync with: 127.0.0.1:8301
2020/01/22 11:52:45 [INFO] serf: EventMemberJoin: wet-fish-consul-server-0 10.1.0.129
2020/01/22 11:52:45 [INFO] serf: EventMemberJoin: docker-for-desktop 10.1.0.128
2020/01/22 11:52:45 [INFO] agent: (LAN) joined: 1
2020/01/22 11:52:45 [DEBUG] agent: systemd notify failed: No socket
2020/01/22 11:52:45 [INFO] agent: Join LAN completed. Synced with 1 initial agents
2020/01/22 11:52:45 [INFO] consul: adding server wet-fish-consul-server-0 (Addr: tcp/10.1.0.129:8300) (DC: k8s)
2020/01/22 11:52:46 [DEBUG] tlsutil: OutgoingRPCWrapper with version 1
2020/01/22 11:52:46 [DEBUG] memberlist: Failed ping: wet-fish-consul-server-0 (timeout reached)
2020/01/22 11:52:47 [INFO] memberlist: Suspect wet-fish-consul-server-0 has failed, no acks received
2020/01/22 11:52:47 [DEBUG] memberlist: Failed ping: docker-for-desktop (timeout reached)
2020/01/22 11:52:48 [WARN] agent: Check "service:graph-api-local-sidecar-proxy:1" socket connection failed: dial tcp 127.0.0.1:21000: connect: connection refused
2020/01/22 11:52:49 [INFO] memberlist: Suspect docker-for-desktop has failed, no acks received
2020/01/22 11:52:49 [DEBUG] memberlist: Failed ping: wet-fish-consul-server-0 (timeout reached)
2020/01/22 11:52:52 [INFO] memberlist: Suspect wet-fish-consul-server-0 has failed, no acks received
2020/01/22 11:52:52 [DEBUG] memberlist: Failed ping: docker-for-desktop (timeout reached)
2020/01/22 11:52:53 [INFO] memberlist: Marking docker-for-desktop as failed, suspect timeout reached (0 peer confirmations)
2020/01/22 11:52:53 [INFO] serf: EventMemberFailed: docker-for-desktop 10.1.0.128
2020/01/22 11:52:54 [DEBUG] memberlist: Stream connection from=192.168.10.222:50442
2020/01/22 11:52:54 [INFO] serf: EventMemberJoin: docker-for-desktop 10.1.0.128
2020/01/22 11:52:56 [INFO] memberlist: Suspect docker-for-desktop has failed, no acks received
2020/01/22 11:52:56 [ERR] consul: "Catalog.NodeServices" RPC failed to server 10.1.0.129:8300: rpc error getting client: failed to get conn: dial tcp <nil>->10.1.0.129:8300: i/o timeout
2020/01/22 11:52:56 [ERR] consul: "ConnectCA.Roots" RPC failed to server 10.1.0.129:8300: rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection
2020/01/22 11:52:56 [ERR] consul: "Intention.Match" RPC failed to server 10.1.0.129:8300: rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection
2020/01/22 11:52:56 [ERR] agent: failed to sync remote state: rpc error getting client: failed to get conn: dial tcp <nil>->10.1.0.129:8300: i/o timeout
2020/01/22 11:52:56 [ERR] intentions watch error: error filling agent cache: rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection
2020/01/22 11:52:56 [ERR] roots watch error: error filling agent cache: rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection
2020/01/22 11:52:56 [ERR] leaf watch error: error filling agent cache: rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection
2020/01/22 11:52:56 [DEBUG] tlsutil: OutgoingRPCWrapper with version 1
2020/01/22 11:52:56 [DEBUG] memberlist: Failed ping: docker-for-desktop (timeout reached)
2020/01/22 11:52:58 [WARN] agent: Check "service:graph-api-local-sidecar-proxy:1" socket connection failed: dial tcp 127.0.0.1:21000: connect: connection refused
2020/01/22 11:53:01 [INFO] memberlist: Suspect docker-for-desktop has failed, no acks received
2020/01/22 11:53:01 [DEBUG] memberlist: Failed ping: wet-fish-consul-server-0 (timeout reached)
2020/01/22 11:53:05 [INFO] memberlist: Marking docker-for-desktop as failed, suspect timeout reached (0 peer confirmations)
2020/01/22 11:53:05 [INFO] serf: EventMemberFailed: docker-for-desktop 10.1.0.128
2020/01/22 11:53:05 [INFO] serf: EventMemberJoin: docker-for-desktop 10.1.0.128
2020/01/22 11:53:06 [ERR] consul: "ConnectCA.Roots" RPC failed to server 10.1.0.129:8300: rpc error getting client: failed to get conn: dial tcp <nil>->10.1.0.129:8300: i/o timeout
2020/01/22 11:53:06 [ERR] consul: "Intention.Match" RPC failed to server 10.1.0.129:8300: rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection
2020/01/22 11:53:06 [ERR] consul: "Coordinate.Update" RPC failed to server 10.1.0.129:8300: rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection
2020/01/22 11:53:06 [ERR] roots watch error: error filling agent cache: rpc error getting client: failed to get conn: dial tcp <nil>->10.1.0.129:8300: i/o timeout
2020/01/22 11:53:06 [DEBUG] tlsutil: OutgoingRPCWrapper with version 1
2020/01/22 11:53:06 [ERR] agent: Coordinate update error: rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection
2020/01/22 11:53:06 [ERR] intentions watch error: error filling agent cache: rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection
2020/01/22 11:53:06 [ERR] leaf watch error: error filling agent cache: rpc error getting client: failed to get conn: dial tcp <nil>->10.1.0.129:8300: i/o timeout
2020/01/22 11:53:07 [INFO] memberlist: Suspect wet-fish-consul-server-0 has failed, no acks received
2020/01/22 11:53:07 [DEBUG] memberlist: Failed ping: wet-fish-consul-server-0 (timeout reached)
2020/01/22 11:53:08 [WARN] agent: Check "service:graph-api-local-sidecar-proxy:1" socket connection failed: dial tcp 127.0.0.1:21000: connect: connection refused
2020/01/22 11:53:11 [INFO] memberlist: Marking wet-fish-consul-server-0 as failed, suspect timeout reached (0 peer confirmations)
2020/01/22 11:53:11 [INFO] serf: EventMemberFailed: wet-fish-consul-server-0 10.1.0.129
2020/01/22 11:53:11 [INFO] consul: removing server wet-fish-consul-server-0 (Addr: tcp/10.1.0.129:8300) (DC: k8s)
2020/01/22 11:53:12 [WARN] manager: No servers available
2020/01/22 11:53:12 [ERR] agent: failed to sync remote state: No known Consul servers
2020/01/22 11:53:14 [INFO] memberlist: Suspect wet-fish-consul-server-0 has failed, no acks received
2020/01/22 11:53:14 [DEBUG] memberlist: Failed ping: docker-for-desktop (timeout reached)
2020/01/22 11:53:15 [INFO] serf: attempting reconnect to wet-fish-consul-server-0 10.1.0.129:8301
2020/01/22 11:53:16 [ERR] consul: "ConnectCA.Roots" RPC failed to server 10.1.0.129:8300: rpc error getting client: failed to get conn: dial tcp <nil>->10.1.0.129:8300: i/o timeout
2020/01/22 11:53:16 [ERR] consul: "Intention.Match" RPC failed to server 10.1.0.129:8300: rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection
2020/01/22 11:53:16 [WARN] manager: No servers available
2020/01/22 11:53:16 [WARN] manager: No servers available
2020/01/22 11:53:16 [ERR] intentions watch error: error filling agent cache: rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection
2020/01/22 11:53:16 [WARN] manager: No servers available
2020/01/22 11:53:16 [WARN] manager: No servers available
2020/01/22 11:53:16 [WARN] manager: No servers available
2020/01/22 11:53:16 [WARN] manager: No servers available
2020/01/22 11:53:18 [WARN] manager: No servers available
2020/01/22 11:53:18 [WARN] manager: No servers available
2020/01/22 11:53:18 [WARN] manager: No servers available
2020/01/22 11:53:18 [WARN] manager: No servers available
2020/01/22 11:53:18 [WARN] manager: No servers available

I also get these logs

2020/01/22 12:49:58 [DEBUG] memberlist: Failed ping: wet-fish-consul-server-0 (timeout reached)
2020/01/22 12:49:58 [WARN] agent: Check "service:graph-api-local-sidecar-proxy:1" socket connection failed: dial tcp 127.0.0.1:21000: connect: connection refused
2020/01/22 12:50:01 [ERR] consul: "Catalog.NodeServices" RPC failed to server 10.1.0.129:8300: rpc error getting client: failed to get conn: dial tcp <nil>->10.1.0.129:8300: i/o timeout
2020/01/22 12:50:01 [ERR] consul: "ConnectCA.Roots" RPC failed to server 10.1.0.129:8300: rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection
2020/01/22 12:50:01 [ERR] agent: failed to sync remote state: rpc error getting client: failed to get conn: dial tcp <nil>->10.1.0.129:8300: i/o timeout
2020/01/22 12:50:01 [ERR] consul: "Intention.Match" RPC failed to server 10.1.0.129:8300: rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection
2020/01/22 12:50:01 [ERR] roots watch error: error filling agent cache: rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection
2020/01/22 12:50:01 [ERR] leaf watch error: error filling agent cache: rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection
2020/01/22 12:50:01 [ERR] intentions watch error: error filling agent cache: rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection
2020/01/22 12:50:01 [DEBUG] tlsutil: OutgoingRPCWrapper with version 1

It definitely looks like not all of the IPs are routable from everything else.