No cluster leader (5 node cluster, how to recover?)

[maxb]

You may be experiencing problems similar to this other active topic:

3-server cluster becomes unstable when follower recovers from temporary network outage?

I’m running into a problem that seems to be occurring independently with both Consul (1.13.1 tested so far) as well as Nomad (1.3.5 tested so far). I have a 3-server cluster, where if either of the non-leader nodes suffers a temporary network outage, the cluster continues to be ok during the outage, as well as afterward – if the outage was fairly short (eg seconds). However, if the outage lasts slightly longer (eg > 30 seconds), then as soon as connectivity resumes, the cluster immediately goe…

But what we really need to see to say for certain is consul operator raft list-peers -stale output from each node, to understand what state the cluster is in. That will requiring finding a suitably permissioned ACL token that already exists on the cluster.