When consul cluster enters into a no leader state, I can still use consul operator list-peers -stale to get all the peer information, but to actually recover i needed to manually create a peers.json file, I’ve done it multiple times with success, but my question is that why couldn’t I just use consul operator remove-peer -stale ? It complains about the cluster have no leader, which make sense, but even with peers.json I only needed to change 1 machine for it to be elected leader.
Supporting -stale on a read operation is easy - it just means to respond from local data, skipping the “am I reasonably up to date” check.
However, the definition of a write operation, is to send the operation to the cluster leader and have it replicate it to a quorum of nodes. It’s simply not possible to do this without a leader, because by definition it involves talking to the leader.
When you do a peers.json recovery, very unusual things are happening: normal properties of the Raft algorithm get bypassed to force a new configuration (set of nodes) into place.
There is an implicit assumption that you will either be performing the peers.json procedure on all nodes of a cluster, or only one, whilst wiping and resetting others from scratch:
If you are somehow managing to use peers.json only on one node of an existing cluster, then I think that’s an unintended way to use it.