Despite the fact that Consul has a docker image I can’t seem to make a 3 node consul service actually work.
I run a 3 node consul cluster with a docker-compose.yml like this:
services:
consul:
image: consul:latest
command: |
agent
-server
-retry-join=tasks.consul
-bootstrap-expect=${CONSUL_REPLICAS-3}
-datacenter=test
-client=0.0.0.0
-ui
deploy:
replicas: 3
Then I kill one of the docker tasks and wait for docker/k8s to restart a new instance. Then runnin (on each of the tasks) consul operator raft list-peers
drops to 2 and doesn’t properly recover, and consul members
shows different numbers of nodes. If I do this a few more times the cluster looses raft consensus and never recovers.