Hello.
I have fresh consul cluster setup with following config:
consul.hcl
datacenter = "dc1"
node_name = "node1"
server = true
rejoin_after_leave = true
retry_join = ["node1.internal", "node2.internal", "node3.internal"]
bind_addr = "192.168.1.1"
advertise_addr = "192.168.1.1"
client_addr = "127.0.0.1 10.1.8.1"
All consul agents run in docker:
docker-compose.yml
version: '3.5'
services:
consul:
image: consul:1.6.0-beta1
container_name: consul
restart: always
network_mode: host
volumes:
- /etc/consul/config:/consul/config
- /var/consul/data:/consul/data
Consul cluster constantly elect new leader and I have absolutely no idea why this happens. I think this is due to raft: Failed to contact warning, but all ports are open and traffic not restricted. I even looked at tcpdump on port 8300 and there was no failed TCP packets at all. Newly elected leader starts writing raft: Failed to contact warning in it’s own log and after short time new election started.
docker logs consul
==> Starting Consul agent...
Version: 'v1.6.0-beta1 (83b929acc)'
Node ID: 'a94553fa-7131-005e-658d-9bdff67fdf86'
Node name: 'node1'
Datacenter: 'dc1' (Segment: '<all>')
Server: true (Bootstrap: false)
Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
Cluster Addr: 192.168.1.1 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
==> Log data will now stream in as it occurs:
2019/08/05 13:14:14 [DEBUG] agent: Using random ID "a94553fa-7131-005e-658d-9bdff67fdf86" as node ID
2019/08/05 13:14:14 [DEBUG] tlsutil: Update with version 1
2019/08/05 13:14:14 [DEBUG] tlsutil: OutgoingRPCWrapper with version 1
2019/08/05 13:14:14 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID:a94553fa-7131-005e-658d-9bdff67fdf86 Address:192.168.1.1:8300}]
2019/08/05 13:14:14 [INFO] raft: Node at 192.168.1.1:8300 [Follower] entering Follower state (Leader: "")
2019/08/05 13:14:14 [INFO] serf: EventMemberJoin: node1.dc1 192.168.1.1
2019/08/05 13:14:14 [INFO] serf: EventMemberJoin: node1 192.168.1.1
2019/08/05 13:14:14 [INFO] consul: Handled member-join event for server "node1.dc1" in area "wan"
2019/08/05 13:14:14 [DEBUG] agent/proxy: managed Connect proxy manager started
2019/08/05 13:14:14 [INFO] consul: Adding LAN server node1 (Addr: tcp/192.168.1.1:8300) (DC: dc1)
2019/08/05 13:14:14 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp)
2019/08/05 13:14:14 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp)
2019/08/05 13:14:14 [INFO] agent: Started HTTP server on [::]:8500 (tcp)
2019/08/05 13:14:14 [INFO] agent: Started gRPC server on [::]:8502 (tcp)
2019/08/05 13:14:14 [INFO] agent: Retry join LAN is supported for: aliyun aws azure digitalocean gce k8s mdns os packet scaleway softlayer triton vsphere
2019/08/05 13:14:14 [INFO] agent: Joining LAN cluster...
2019/08/05 13:14:14 [INFO] agent: (LAN) joining: [node3.internal node2.internal node1.internal]
2019/08/05 13:14:14 [INFO] agent: started state syncer
==> Consul agent running!
2019/08/05 13:14:14 [DEBUG] memberlist: TCP-first lookup failed for 'node3.internal:8301', falling back to UDP: dns: buffer size too small
2019/08/05 13:14:14 [DEBUG] memberlist: Failed to join 192.168.1.3: dial tcp 192.168.1.3:8301: connect: connection refused
2019/08/05 13:14:14 [DEBUG] memberlist: TCP-first lookup failed for 'node2.internal:8301', falling back to UDP: dns: buffer size too small
2019/08/05 13:14:14 [DEBUG] memberlist: Failed to join 192.168.1.2: dial tcp 192.168.1.2:8301: connect: connection refused
2019/08/05 13:14:14 [DEBUG] memberlist: TCP-first lookup failed for 'node1.internal:8301', falling back to UDP: dns: id mismatch
2019/08/05 13:14:14 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.1:8301
2019/08/05 13:14:14 [DEBUG] memberlist: Stream connection from=192.168.1.1:45854
2019/08/05 13:14:14 [INFO] agent: (LAN) joined: 1
2019/08/05 13:14:14 [DEBUG] agent: systemd notify failed: No socket
2019/08/05 13:14:14 [INFO] agent: Join LAN completed. Synced with 1 initial agents
2019/08/05 13:14:14 [WARN] raft: Heartbeat timeout from "" reached, starting election
2019/08/05 13:14:14 [INFO] raft: Node at 192.168.1.1:8300 [Candidate] entering Candidate state in term 2
2019/08/05 13:14:14 [DEBUG] raft: Votes needed: 1
2019/08/05 13:14:14 [DEBUG] raft: Vote granted from a94553fa-7131-005e-658d-9bdff67fdf86 in term 2. Tally: 1
2019/08/05 13:14:14 [INFO] raft: Election won. Tally: 1
2019/08/05 13:14:14 [INFO] raft: Node at 192.168.1.1:8300 [Leader] entering Leader state
2019/08/05 13:14:14 [INFO] consul: cluster leadership acquired
2019/08/05 13:14:14 [INFO] consul: New leader elected: node1
2019/08/05 13:14:14 [INFO] connect: initialized primary datacenter CA with provider "consul"
2019/08/05 13:14:14 [DEBUG] consul: Skipping self join check for "node1" since the cluster is too small
2019/08/05 13:14:14 [INFO] consul: member 'node1' joined, marking health alive
2019/08/05 13:14:14 [DEBUG] agent: Skipping remote check "serfHealth" since it is managed automatically
2019/08/05 13:14:14 [INFO] agent: Synced node info
2019/08/05 13:14:16 [DEBUG] tlsutil: OutgoingRPCWrapper with version 1
2019/08/05 13:14:16 [DEBUG] memberlist: Stream connection from=192.168.1.2:58188
2019/08/05 13:14:16 [INFO] serf: EventMemberJoin: node2 192.168.1.2
2019/08/05 13:14:16 [INFO] consul: Adding LAN server node2 (Addr: tcp/192.168.1.2:8300) (DC: dc1)
2019/08/05 13:14:16 [INFO] raft: Updating configuration with AddNonvoter (b71f41d0-4b00-5958-0098-f8bcfecb6489, 192.168.1.2:8300) to [{Suffrage:Voter ID:a94553fa-7131-005e-658d-9bdff67fdf86 Address:192.168.1.1:8300} {Suffrage:Nonvoter ID:b71f41d0-4b00-5958-0098-f8bcfecb6489 Address:192.168.1.2:8300}]
2019/08/05 13:14:16 [INFO] raft: Added peer b71f41d0-4b00-5958-0098-f8bcfecb6489, starting replication
2019/08/05 13:14:16 [INFO] consul: member 'node2' joined, marking health alive
2019/08/05 13:14:16 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.2:8302
2019/08/05 13:14:16 [WARN] raft: AppendEntries to {Nonvoter b71f41d0-4b00-5958-0098-f8bcfecb6489 192.168.1.2:8300} rejected, sending older logs (next: 2)
2019/08/05 13:14:16 [INFO] serf: EventMemberJoin: node2.dc1 192.168.1.2
2019/08/05 13:14:16 [DEBUG] memberlist: Stream connection from=192.168.1.2:59300
2019/08/05 13:14:16 [DEBUG] consul: Successfully performed flood-join for "node2" at 192.168.1.2:8302
2019/08/05 13:14:16 [INFO] consul: Handled member-join event for server "node2.dc1" in area "wan"
2019/08/05 13:14:16 [INFO] raft: pipelining replication to peer {Nonvoter b71f41d0-4b00-5958-0098-f8bcfecb6489 192.168.1.2:8300}
2019/08/05 13:14:16 [DEBUG] serf: messageJoinType: node2.dc1
2019/08/05 13:14:16 [DEBUG] serf: messageJoinType: node2
2019/08/05 13:14:16 [DEBUG] serf: messageJoinType: node2.dc1
2019/08/05 13:14:16 [DEBUG] serf: messageJoinType: node2.dc1
2019/08/05 13:14:16 [DEBUG] serf: messageJoinType: node2
2019/08/05 13:14:16 [DEBUG] serf: messageJoinType: node2
2019/08/05 13:14:16 [DEBUG] serf: messageJoinType: node2.dc1
2019/08/05 13:14:16 [DEBUG] serf: messageJoinType: node2
2019/08/05 13:14:17 [DEBUG] agent: Skipping remote check "serfHealth" since it is managed automatically
2019/08/05 13:14:17 [DEBUG] agent: Node info in sync
2019/08/05 13:14:17 [DEBUG] agent: Node info in sync
2019/08/05 13:14:18 [DEBUG] tlsutil: OutgoingRPCWrapper with version 1
2019/08/05 13:14:19 [DEBUG] memberlist: Stream connection from=192.168.1.3:35850
2019/08/05 13:14:19 [INFO] serf: EventMemberJoin: node3 192.168.1.3
2019/08/05 13:14:19 [INFO] consul: Adding LAN server node3 (Addr: tcp/192.168.1.3:8300) (DC: dc1)
2019/08/05 13:14:19 [DEBUG] memberlist: Stream connection from=192.168.1.3:32998
2019/08/05 13:14:19 [INFO] raft: Updating configuration with AddNonvoter (a8304563-736b-11c0-1044-9d57e817d6d1, 192.168.1.3:8300) to [{Suffrage:Voter ID:a94553fa-7131-005e-658d-9bdff67fdf86 Address:192.168.1.1:8300} {Suffrage:Nonvoter ID:b71f41d0-4b00-5958-0098-f8bcfecb6489 Address:192.168.1.2:8300} {Suffrage:Nonvoter ID:a8304563-736b-11c0-1044-9d57e817d6d1 Address:192.168.1.3:8300}]
2019/08/05 13:14:19 [INFO] raft: Added peer a8304563-736b-11c0-1044-9d57e817d6d1, starting replication
2019/08/05 13:14:19 [INFO] consul: member 'node3' joined, marking health alive
2019/08/05 13:14:19 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.3:8302
2019/08/05 13:14:19 [INFO] serf: EventMemberJoin: node3.dc1 192.168.1.3
2019/08/05 13:14:19 [INFO] consul: Handled member-join event for server "node3.dc1" in area "wan"
2019/08/05 13:14:19 [WARN] raft: AppendEntries to {Nonvoter a8304563-736b-11c0-1044-9d57e817d6d1 192.168.1.3:8300} rejected, sending older logs (next: 2)
2019/08/05 13:14:19 [INFO] raft: pipelining replication to peer {Nonvoter a8304563-736b-11c0-1044-9d57e817d6d1 192.168.1.3:8300}
2019/08/05 13:14:19 [DEBUG] consul: Successfully performed flood-join for "node3" at 192.168.1.3:8302
2019/08/05 13:14:19 [DEBUG] serf: messageJoinType: node3.dc1
2019/08/05 13:14:19 [DEBUG] serf: messageJoinType: node3.dc1
2019/08/05 13:14:19 [DEBUG] serf: messageJoinType: node2.dc1
2019/08/05 13:14:19 [DEBUG] serf: messageJoinType: node1.dc1
2019/08/05 13:14:19 [DEBUG] serf: messageJoinType: node3.dc1
2019/08/05 13:14:19 [DEBUG] serf: messageJoinType: node3.dc1
2019/08/05 13:14:19 [DEBUG] serf: messageJoinType: node1.dc1
2019/08/05 13:14:19 [DEBUG] serf: messageJoinType: node2.dc1
2019/08/05 13:14:19 [DEBUG] serf: messageJoinType: node3
2019/08/05 13:14:19 [DEBUG] serf: messageJoinType: node2.dc1
2019/08/05 13:14:19 [DEBUG] serf: messageJoinType: node1.dc1
2019/08/05 13:14:19 [DEBUG] serf: messageJoinType: node3.dc1
2019/08/05 13:14:19 [DEBUG] serf: messageJoinType: node3.dc1
2019/08/05 13:14:19 [DEBUG] serf: messageJoinType: node3
2019/08/05 13:14:19 [DEBUG] serf: messageJoinType: node3
2019/08/05 13:14:20 [DEBUG] tlsutil: OutgoingRPCWrapper with version 1
2019/08/05 13:14:34 [INFO] autopilot: Promoting Server (ID: "b71f41d0-4b00-5958-0098-f8bcfecb6489" Address: "192.168.1.2:8300") to voter
2019/08/05 13:14:34 [INFO] raft: Updating configuration with AddStaging (b71f41d0-4b00-5958-0098-f8bcfecb6489, 192.168.1.2:8300) to [{Suffrage:Voter ID:a94553fa-7131-005e-658d-9bdff67fdf86 Address:192.168.1.1:8300} {Suffrage:Voter ID:b71f41d0-4b00-5958-0098-f8bcfecb6489 Address:192.168.1.2:8300} {Suffrage:Nonvoter ID:a8304563-736b-11c0-1044-9d57e817d6d1 Address:192.168.1.3:8300}]
2019/08/05 13:14:34 [INFO] autopilot: Promoting Server (ID: "a8304563-736b-11c0-1044-9d57e817d6d1" Address: "192.168.1.3:8300") to voter
2019/08/05 13:14:34 [INFO] raft: Updating configuration with AddStaging (a8304563-736b-11c0-1044-9d57e817d6d1, 192.168.1.3:8300) to [{Suffrage:Voter ID:a94553fa-7131-005e-658d-9bdff67fdf86 Address:192.168.1.1:8300} {Suffrage:Voter ID:b71f41d0-4b00-5958-0098-f8bcfecb6489 Address:192.168.1.2:8300} {Suffrage:Voter ID:a8304563-736b-11c0-1044-9d57e817d6d1 Address:192.168.1.3:8300}]
2019/08/05 13:14:44 [DEBUG] memberlist: Stream connection from=192.168.1.8:40876
2019/08/05 13:14:44 [INFO] serf: EventMemberJoin: node8 192.168.1.8
2019/08/05 13:14:44 [INFO] consul: member 'node8' joined, marking health alive
2019/08/05 13:14:44 [DEBUG] serf: messageJoinType: node8
2019/08/05 13:14:44 [DEBUG] serf: messageJoinType: node8
2019/08/05 13:14:44 [DEBUG] serf: messageJoinType: node8
2019/08/05 13:14:44 [DEBUG] serf: messageJoinType: node8
2019/08/05 13:14:47 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 28.757494ms
2019/08/05 13:14:47 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 42.702251ms
2019/08/05 13:14:53 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 25.026068ms
2019/08/05 13:14:59 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.2:8301
2019/08/05 13:15:01 [DEBUG] memberlist: Stream connection from=192.168.1.3:36846
2019/08/05 13:15:05 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 20.280223ms
2019/08/05 13:15:16 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 20.273001ms
2019/08/05 13:15:25 [DEBUG] memberlist: Stream connection from=192.168.1.2:60176
2019/08/05 13:15:29 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.8:8301
2019/08/05 13:15:31 [DEBUG] memberlist: Stream connection from=192.168.1.3:51278
2019/08/05 13:15:40 [DEBUG] http: Request GET /v1/operator/raft/configuration (725.738µs) from=127.0.0.1:48264
2019/08/05 13:15:43 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.2:8302
2019/08/05 13:15:49 [DEBUG] http: Request GET /v1/agent/members?segment=_all (1.469544ms) from=127.0.0.1:48280
2019/08/05 13:15:53 [DEBUG] agent: Skipping remote check "serfHealth" since it is managed automatically
2019/08/05 13:15:53 [DEBUG] agent: Node info in sync
2019/08/05 13:15:59 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.8:8301
2019/08/05 13:16:01 [DEBUG] memberlist: Stream connection from=192.168.1.3:48252
2019/08/05 13:16:01 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 20.547876ms
2019/08/05 13:16:01 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 35.159626ms
2019/08/05 13:16:01 [DEBUG] memberlist: Stream connection from=192.168.1.8:41224
2019/08/05 13:16:07 [DEBUG] memberlist: Stream connection from=192.168.1.2:59598
2019/08/05 13:16:14 [DEBUG] manager: Rebalanced 3 servers, next active server is node1.dc1 (Addr: tcp/192.168.1.1:8300) (DC: dc1)
2019/08/05 13:16:15 [DEBUG] memberlist: Stream connection from=192.168.1.3:47358
2019/08/05 13:16:25 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 20.258679ms
2019/08/05 13:16:25 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 39.680911ms
2019/08/05 13:16:25 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 25.641049ms
2019/08/05 13:16:29 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.2:8301
2019/08/05 13:16:31 [DEBUG] memberlist: Stream connection from=192.168.1.3:51826
2019/08/05 13:16:37 [DEBUG] memberlist: Stream connection from=192.168.1.2:59994
2019/08/05 13:16:43 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.2:8302
2019/08/05 13:16:55 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 24.716774ms
2019/08/05 13:16:55 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 40.818292ms
2019/08/05 13:16:59 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.8:8301
2019/08/05 13:17:01 [DEBUG] memberlist: Stream connection from=192.168.1.3:32904
2019/08/05 13:17:01 [DEBUG] memberlist: Stream connection from=192.168.1.8:41602
2019/08/05 13:17:05 [DEBUG] agent: Skipping remote check "serfHealth" since it is managed automatically
2019/08/05 13:17:05 [DEBUG] agent: Node info in sync
2019/08/05 13:17:28 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 20.258193ms
2019/08/05 13:17:28 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 37.82671ms
2019/08/05 13:17:29 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.3:8301
2019/08/05 13:17:31 [DEBUG] memberlist: Stream connection from=192.168.1.3:53172
2019/08/05 13:17:31 [DEBUG] memberlist: Stream connection from=192.168.1.8:41694
2019/08/05 13:17:37 [DEBUG] memberlist: Stream connection from=192.168.1.2:60794
2019/08/05 13:17:43 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.3:8302
2019/08/05 13:17:56 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 20.145379ms
2019/08/05 13:17:57 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 20.259845ms
2019/08/05 13:17:59 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.8:8301
2019/08/05 13:18:01 [DEBUG] memberlist: Stream connection from=192.168.1.8:41788
2019/08/05 13:18:06 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 20.237152ms
2019/08/05 13:18:07 [DEBUG] memberlist: Stream connection from=192.168.1.2:32998
2019/08/05 13:18:12 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 28.424514ms
2019/08/05 13:18:12 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 42.522308ms
2019/08/05 13:18:20 [DEBUG] agent: Skipping remote check "serfHealth" since it is managed automatically
2019/08/05 13:18:20 [DEBUG] agent: Node info in sync
2019/08/05 13:18:21 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 20.206952ms
2019/08/05 13:18:25 [DEBUG] memberlist: Stream connection from=192.168.1.2:34344
2019/08/05 13:18:29 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.3:8301
2019/08/05 13:18:34 [DEBUG] manager: Rebalanced 3 servers, next active server is node3.dc1 (Addr: tcp/192.168.1.3:8300) (DC: dc1)
2019/08/05 13:18:43 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.2:8302
2019/08/05 13:18:59 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.2:8301
2019/08/05 13:19:05 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 23.045889ms
2019/08/05 13:19:08 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 24.377156ms
2019/08/05 13:19:14 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 25.704768ms
2019/08/05 13:19:14 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 40.473034ms
2019/08/05 13:19:16 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 25.289313ms
2019/08/05 13:19:17 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 23.218614ms
2019/08/05 13:19:21 [DEBUG] agent: Skipping remote check "serfHealth" since it is managed automatically
2019/08/05 13:19:21 [DEBUG] agent: Node info in sync
2019/08/05 13:19:25 [DEBUG] memberlist: Stream connection from=192.168.1.2:35186
2019/08/05 13:19:25 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 20.152186ms
2019/08/05 13:19:29 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.3:8301
2019/08/05 13:19:31 [DEBUG] memberlist: Stream connection from=192.168.1.3:52984
2019/08/05 13:19:31 [DEBUG] memberlist: Stream connection from=192.168.1.8:42204
2019/08/05 13:19:43 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.2:8302
2019/08/05 13:19:45 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 20.266405ms
2019/08/05 13:19:45 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 40.07465ms
2019/08/05 13:19:54 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 21.48181ms
2019/08/05 13:20:00 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.8:8301
2019/08/05 13:20:04 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 29.111207ms
2019/08/05 13:20:04 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 44.722459ms
2019/08/05 13:20:27 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 20.200252ms
2019/08/05 13:20:30 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.2:8301
2019/08/05 13:20:43 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.3:8302
2019/08/05 13:20:45 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 20.275457ms
2019/08/05 13:20:46 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 20.27658ms
2019/08/05 13:20:46 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 35.616384ms
2019/08/05 13:20:46 [DEBUG] agent: Skipping remote check "serfHealth" since it is managed automatically
2019/08/05 13:20:46 [DEBUG] agent: Node info in sync
2019/08/05 13:20:55 [DEBUG] manager: Rebalanced 3 servers, next active server is node2.dc1 (Addr: tcp/192.168.1.2:8300) (DC: dc1)
2019/08/05 13:21:00 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.8:8301
2019/08/05 13:21:07 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 20.159372ms
2019/08/05 13:21:15 [DEBUG] memberlist: Stream connection from=192.168.1.3:59616
2019/08/05 13:21:25 [DEBUG] memberlist: Stream connection from=192.168.1.2:36938
2019/08/05 13:21:30 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.8:8301
2019/08/05 13:21:43 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.3:8302
2019/08/05 13:21:45 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 20.275733ms
2019/08/05 13:21:51 [DEBUG] agent: Skipping remote check "serfHealth" since it is managed automatically
2019/08/05 13:21:51 [DEBUG] agent: Node info in sync
2019/08/05 13:21:56 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 20.144377ms
2019/08/05 13:22:00 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.2:8301
2019/08/05 13:22:01 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 20.265802ms
2019/08/05 13:22:01 [DEBUG] memberlist: Stream connection from=192.168.1.3:49874
2019/08/05 13:22:01 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 35.192293ms
2019/08/05 13:22:07 [DEBUG] memberlist: Stream connection from=192.168.1.2:36524
2019/08/05 13:22:15 [DEBUG] memberlist: Stream connection from=192.168.1.3:38312
2019/08/05 13:22:17 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 20.202275ms
2019/08/05 13:22:25 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 22.493029ms
2019/08/05 13:22:29 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 24.659663ms
2019/08/05 13:22:29 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 27.171145ms
2019/08/05 13:22:30 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.3:8301
2019/08/05 13:22:31 [DEBUG] memberlist: Stream connection from=192.168.1.3:38450
2019/08/05 13:22:31 [DEBUG] memberlist: Stream connection from=192.168.1.8:43338
2019/08/05 13:22:36 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 20.196342ms
2019/08/05 13:22:43 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.3:8302
2019/08/05 13:22:46 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 23.439596ms
2019/08/05 13:22:47 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 21.549794ms
2019/08/05 13:22:47 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 38.90346ms
2019/08/05 13:22:57 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 20.961537ms
2019/08/05 13:22:57 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 23.116448ms
2019/08/05 13:22:57 [WARN] raft: Failed to contact a8304563-736b-11c0-1044-9d57e817d6d1 in 40.946944ms
2019/08/05 13:22:57 [DEBUG] raft-net: 192.168.1.1:8300 accepted connection from: 192.168.1.3:33911
2019/08/05 13:22:57 [WARN] raft: Rejecting vote request from 192.168.1.3:8300 since we have a leader: 192.168.1.1:8300
2019/08/05 13:22:57 [ERROR] raft: peer {Nonvoter a8304563-736b-11c0-1044-9d57e817d6d1 192.168.1.3:8300} has newer term, stopping replication
2019/08/05 13:22:57 [INFO] raft: aborting pipeline replication to peer {Nonvoter a8304563-736b-11c0-1044-9d57e817d6d1 192.168.1.3:8300}
2019/08/05 13:22:57 [INFO] raft: Node at 192.168.1.1:8300 [Follower] entering Follower state (Leader: "")
2019/08/05 13:22:57 [DEBUG] consul: shutting down leader loop
2019/08/05 13:22:57 [INFO] raft: aborting pipeline replication to peer {Nonvoter b71f41d0-4b00-5958-0098-f8bcfecb6489 192.168.1.2:8300}
2019/08/05 13:22:57 [INFO] consul: cluster leadership lost
2019/08/05 13:22:57 [DEBUG] raft: lost leadership because received a requestvote with newer term
2019/08/05 13:22:57 [DEBUG] raft-net: 192.168.1.1:8300 accepted connection from: 192.168.1.3:37605
2019/08/05 13:22:57 [DEBUG] serf: messageUserEventType: consul:new-leader
2019/08/05 13:22:57 [INFO] consul: New leader elected: node3
2019/08/05 13:22:57 [DEBUG] serf: messageUserEventType: consul:new-leader
2019/08/05 13:22:57 [DEBUG] serf: messageUserEventType: consul:new-leader
2019/08/05 13:23:00 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.2:8301
2019/08/05 13:23:00 [DEBUG] agent: Skipping remote check "serfHealth" since it is managed automatically
2019/08/05 13:23:00 [DEBUG] agent: Node info in sync
2019/08/05 13:23:01 [DEBUG] memberlist: Stream connection from=192.168.1.8:43492
2019/08/05 13:23:11 [DEBUG] manager: Rebalanced 3 servers, next active server is node1.dc1 (Addr: tcp/192.168.1.1:8300) (DC: dc1)
2019/08/05 13:23:30 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.3:8301
2019/08/05 13:23:31 [DEBUG] memberlist: Stream connection from=192.168.1.8:43666
2019/08/05 13:23:43 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.3:8302
2019/08/05 13:24:00 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.2:8301
2019/08/05 13:24:01 [DEBUG] memberlist: Stream connection from=192.168.1.3:44076
2019/08/05 13:24:07 [DEBUG] memberlist: Stream connection from=192.168.1.2:38244
2019/08/05 13:24:15 [DEBUG] memberlist: Stream connection from=192.168.1.3:42096
2019/08/05 13:24:20 [DEBUG] agent: Skipping remote check "serfHealth" since it is managed automatically
2019/08/05 13:24:20 [DEBUG] agent: Node info in sync
2019/08/05 13:24:25 [DEBUG] memberlist: Stream connection from=192.168.1.2:39634
2019/08/05 13:24:30 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.3:8301
2019/08/05 13:24:31 [DEBUG] memberlist: Stream connection from=192.168.1.3:36444
2019/08/05 13:24:37 [DEBUG] memberlist: Stream connection from=192.168.1.2:38678
2019/08/05 13:24:43 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.3:8302
2019/08/05 13:25:00 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.8:8301
2019/08/05 13:25:15 [DEBUG] memberlist: Stream connection from=192.168.1.3:50714
2019/08/05 13:25:18 [DEBUG] tlsutil: OutgoingRPCWrapper with version 1
2019/08/05 13:25:18 [DEBUG] manager: Rebalanced 3 servers, next active server is node1.dc1 (Addr: tcp/192.168.1.1:8300) (DC: dc1)
2019/08/05 13:25:30 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.1.8:8301
2019/08/05 13:25:38 [DEBUG] raft-net: 192.168.1.1:8300 accepted connection from: 192.168.1.2:46981
2019/08/05 13:25:38 [WARN] raft: Rejecting vote request from 192.168.1.2:8300 since we have a leader: 192.168.1.3:8300
2019/08/05 13:25:38 [WARN] raft: Rejecting vote request from 192.168.1.2:8300 since we have a leader: 192.168.1.3:8300
2019/08/05 13:25:38 [DEBUG] raft-net: 192.168.1.1:8300 accepted connection from: 192.168.1.2:43921
2019/08/05 13:25:38 [DEBUG] serf: messageUserEventType: consul:new-leader
2019/08/05 13:25:38 [INFO] consul: New leader elected: node2
2019/08/05 13:25:38 [DEBUG] serf: messageUserEventType: consul:new-leader
2019/08/05 13:25:38 [DEBUG] serf: messageUserEventType: consul:new-leader
2019/08/05 13:25:38 [DEBUG] serf: messageUserEventType: consul:new-leader
2019/08/05 13:25:38 [DEBUG] serf: messageUserEventType: consul:new-leader
2019/08/05 13:25:38 [DEBUG] serf: messageUserEventType: consul:new-leader
consul raft peers and members
# consul members
Node Address Status Type Build Protocol DC Segment
node3 192.168.1.3:8301 alive server 1.6.0beta1 2 dc1 <all>
node2 192.168.1.2:8301 alive server 1.6.0beta1 2 dc1 <all>
node1 192.168.1.1:8301 alive server 1.6.0beta1 2 dc1 <all>
node8 192.168.1.8:8301 alive client 1.6.0beta1 2 dc1 <default>
# consul operator raft list-peers
Node ID Address State Voter RaftProtocol
node1 a94553fa-7131-005e-658d-9bdff67fdf86 192.168.1.1:8300 follower true 3
node2 b71f41d0-4b00-5958-0098-f8bcfecb6489 192.168.1.2:8300 leader true 3
node3 a8304563-736b-11c0-1044-9d57e817d6d1 192.168.1.3:8300 follower true 3
Also why client ports are bound to 0.0.0.0 despite restrictions in client_addr option?