Using Consul with Docker Swarm Mode

Consul
1

Hi everyone!

I’d like to learn how to implement Consul in an environment, which consists of partially Host-Native services and partially services, running in Docker Swarm Mode - Stacks.

We have a total of 6 nodes in our setup with 3 (named “core”) running:

  • Consul Servers
  • Docker Swarm Managers
  • Services like LDAP, Kafka, etc.

and 3 (named “dsm”) running:

  • Consul Clients
  • Docker Swarm Workers

So far, registering the native-host services (LDAP and Kafka so far) on the “core” nodes to Consul with Service Health-Checks works fine. Clients can resolve ldap.service.consul and kafka.service.consul just fine and Consul responds with available services only.

But next, we want to start the first services in Docker Swarm Mode. And here is, where it becomes tricky and I don’t know what’s a working concept:
Containers which are started in the Docker Swarm context, do have some properties, which make them unreachable for Consul Agents running in the Non-Docker Host context of the Docker Swarm Mode nodes:

  • Ports are not exposed to the Host, since they are using Docker overlay networks with swarm scope so only other containers, attached to the same overlay networks, can reach them. This makes it impossible for the Consul Agent to make any Health-Check on the service.
  • IPs to be registered in Consul for a service will be Docker Network internal, private IPs (such as 10.0.2.9/24 for example) which is unrelated and unreachable from the Host-OS or any external node.

If the Consul-Agents are deployed to the Docker Swarm Mode Workers as a Docker container instead of Consul running natively in the Host-OS, it appears to be problematic the other way round:
The problem of resolving the IPs or reaching the other Docker-Service ports will be gone, but others will be introduced:

  • The Consul Containers will have different IPs everytime they are re-created, which would (I assume) cause all kinds of issues with the need to re-register the agents everytime, pile up a ton of dead clients in the Consul members list, etc.
  • Consul Connect won’t work with Host-Native Services and Docker-Container services, appropriately.

So - I hope I expressed the issues I see clearly. Can anyone please try to point me into the right direction? At least concept-wise …