Last thursday we had a weird issue with our Consul cluster. We lost our cluster for a period of time. At the moment we haven’t found any network issues related to this issue outside the nodes. What I did see is that all servers lost communication with the leader, even the leader itself.(!) (consul01 and consul03 where followers and consul02 leader)
Apr 20 14:36:11 consul01 consul[25915]: 2023-04-20T14:36:11.006+0200 [WARN] agent.server.raft: heartbeat timeout reached, starting election: last-leader-addr=10.20.30.38:8300 last-leader-id=0f58059e-23fb-5786-523c-0edbacc1d353
Apr 20 14:36:11 consul01 consul[25915]: agent.server.raft: heartbeat timeout reached, starting election: last-leader-addr=10.20.30.38:8300 last-leader-id=0f58059e-23fb-5786-523c-0edbacc1d353
..
Apr 20 14:36:23 consul02 consul[18384]: 2023-04-20T14:36:23.120+0200 [WARN] agent.server.raft: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
Apr 20 14:36:23 consul02 consul[18384]: 2023-04-20T14:36:23.120+0200 [INFO] agent.server.raft: entering candidate state: node="Node at 10.20.30.38:8300 [Candidate]" term=114895
Apr 20 14:36:23 consul02 consul[18384]: agent.server.raft: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
..
Apr 20 14:36:21 consul03 consul[12664]: 2023-04-20T14:36:21.657+0200 [WARN] agent.server.raft: heartbeat timeout reached, starting election: last-leader-addr=10.20.30.38:8300 last-leader-id=0f58059e-23fb-5786-523c-0edbacc1d353
Apr 20 14:36:21 consul03 consul[12664]: agent.server.raft: heartbeat timeout reached, starting election: last-leader-addr=10.20.30.38:8300 last-leader-id=0f58059e-23fb-5786-523c-0edbacc1d353
Around the same time I also noticed the following message on consul01 and consul03:
Apr 20 14:36:21 consul03 consul[12664]: 2023-04-20T14:36:21.659+0200 [WARN] agent: [core][Channel #14 SubChannel #200825] grpc: addrConn.createTransport failed to connect to {
Apr 20 14:36:21 consul03 consul[12664]: "Addr": "gm-",
Apr 20 14:36:21 consul03 consul[12664]: "ServerName": "leader",
Apr 20 14:36:21 consul03 consul[12664]: "Attributes": null,
Apr 20 14:36:21 consul03 consul[12664]: "BalancerAttributes": null,
Apr 20 14:36:21 consul03 consul[12664]: "Type": 0,
Apr 20 14:36:21 consul03 consul[12664]: "Metadata": null
Apr 20 14:36:21 consul03 consul[12664]: }. Err: connection error: desc = "transport: Error while dialing failed to find Consul server for global address \"gm-\""
Apr 20 14:36:21 consul03 consul[12664]: agent: [core][Channel #14 SubChannel #200825] grpc: addrConn.createTransport failed to connect to {#012 "Addr": "gm-",#012 "ServerName": "leader",#012 "Attributes": null,#012 "BalancerAttributes": null,#012 "Type": 0,#012 "Metadata": null#012}. Err: connection error: desc = "transport: Error while dialing failed to find Consul server for global address \"gm-\""
I cannot find where both servers got this information from. It is not available in the Consul server configuration.