Dear community,
I have noticed recently that we are receiving disconnections in our product.
We are using Consul v0.6.3.
I checked the logs and I noticed that there are too many re-elections and timeouts, which seems to be strange in case if the cluster is stable and there is no reason to failover.
[root@stg-con4 ~]# less /var/log/messages.2 | grep “Apr 28” | grep “starting election”
Apr 28 00:48:56 stg-con4 consul: 2021/04/28 00:48:56 [WARN] raft: Heartbeat timeout reached, starting election
Apr 28 00:48:57 stg-con4 consul: 2021/04/28 00:48:57 [WARN] raft: Election timeout reached, restarting election
Apr 28 04:00:26 stg-con4 consul: 2021/04/28 04:00:26 [WARN] raft: Heartbeat timeout reached, starting election
Apr 28 04:00:27 stg-con4 consul: 2021/04/28 04:00:27 [WARN] raft: Election timeout reached, restarting election
Apr 28 04:31:57 stg-con4 consul: 2021/04/28 04:31:57 [WARN] raft: Heartbeat timeout reached, starting election
Apr 28 04:31:59 stg-con4 consul: 2021/04/28 04:31:59 [WARN] raft: Election timeout reached, restarting election
Apr 28 07:12:56 stg-con4 consul: 2021/04/28 07:12:56 [WARN] raft: Heartbeat timeout reached, starting election
Apr 28 07:12:58 stg-con4 consul: 2021/04/28 07:12:58 [WARN] raft: Election timeout reached, restarting election
Apr 28 10:23:56 stg-con4 consul: 2021/04/28 10:23:56 [WARN] raft: Heartbeat timeout reached, starting election
Apr 28 10:23:58 stg-con4 consul: 2021/04/28 10:23:58 [WARN] raft: Election timeout reached, restarting election
Apr 28 10:38:58 stg-con4 consul: 2021/04/28 10:38:58 [WARN] raft: Heartbeat timeout reached, starting election
Apr 28 10:38:59 stg-con4 consul: 2021/04/28 10:38:59 [WARN] raft: Election timeout reached, restarting election
Apr 28 11:27:04 stg-con4 consul: 2021/04/28 11:27:04 [WARN] raft: Heartbeat timeout reached, starting election
Apr 28 11:27:06 stg-con4 consul: 2021/04/28 11:27:06 [WARN] raft: Election timeout reached, restarting election
Apr 28 17:00:49 stg-con4 consul: 2021/04/28 17:00:49 [WARN] raft: Heartbeat timeout reached, starting election
Apr 28 17:00:51 stg-con4 consul: 2021/04/28 17:00:51 [WARN] raft: Election timeout reached, restarting election
So if you see the whole process of election, goes like this:
Apr 28 17:00:49 stg-con4 consul: 2021/04/28 17:00:49 [WARN] raft: Heartbeat timeout reached, starting election
Apr 28 17:00:49 stg-con4 consul: 2021/04/28 17:00:49 [INFO] raft: Node at xx.xx.0.93:8300 [Candidate] entering Candidate state
Apr 28 17:00:50 stg-con4 consul: 2021/04/28 17:00:50 [INFO] memberlist: Suspect stg-agg1.c.teridion-stg.internal has failed, no acks received
Apr 28 17:00:50 stg-con4 consul: 2021/04/28 17:00:50 [WARN] memberlist: Refuting a suspect message (from: stg-sflow-1.c.teridion-stg.internal)
Apr 28 17:00:51 stg-con4 consul: 2021/04/28 17:00:51 [WARN] raft: Election timeout reached, restarting election
Apr 28 17:00:51 stg-con4 consul: 2021/04/28 17:00:51 [INFO] raft: Node at xx.xx.0.93:8300 [Candidate] entering Candidate state
Apr 28 17:00:51 stg-con4 consul: 2021/04/28 17:00:51 [INFO] raft: Node at xx.xx.0.93:8300 [Follower] entering Follower state
Apr 28 17:00:51 stg-con4 consul: 2021/04/28 17:00:51 [WARN] raft: Failed to get previous log: 20930069 log not found (last: 20930068)
Apr 28 17:00:51 stg-con4 consul: 2021/04/28 17:00:51 [INFO] consul: New leader elected: stg-con2.c.teridion-stg.internal