Using consul operator remove-peer when cluster has no leader

thefallentree · January 4, 2023, 7:58pm

Hello there,

When consul cluster enters into a no leader state, I can still use consul operator list-peers -stale to get all the peer information, but to actually recover i needed to manually create a peers.json file, I’ve done it multiple times with success, but my question is that why couldn’t I just use consul operator remove-peer -stale ? It complains about the cluster have no leader, which make sense, but even with peers.json I only needed to change 1 machine for it to be elected leader.

Thanks

maxb · January 4, 2023, 11:45pm

Supporting -stale on a read operation is easy - it just means to respond from local data, skipping the “am I reasonably up to date” check.

However, the definition of a write operation, is to send the operation to the cluster leader and have it replicate it to a quorum of nodes. It’s simply not possible to do this without a leader, because by definition it involves talking to the leader.

When you do a peers.json recovery, very unusual things are happening: normal properties of the Raft algorithm get bypassed to force a new configuration (set of nodes) into place.

github.com

hashicorp/raft/blob/6b4e32088e0bda22ea219fc89b0ee47f420e2b0b/api.go#L272-L286


      
          // RecoverCluster is used to manually force a new configuration in order to
          // recover from a loss of quorum where the current configuration cannot be
          // restored, such as when several servers die at the same time. This works by
          // reading all the current state for this server, creating a snapshot with the
          // supplied configuration, and then truncating the Raft log. This is the only
          // safe way to force a given configuration without actually altering the log to
          // insert any new entries, which could cause conflicts with other servers with
          // different state.
          //
          // WARNING! This operation implicitly commits all entries in the Raft log, so
          // in general this is an extremely unsafe operation. If you've lost your other
          // servers and are performing a manual recovery, then you've also lost the
          // commit information, so this is likely the best you can do, but you should be
          // aware that calling this can cause Raft log entries that were in the process
          // of being replicated but not yet be committed to be committed.

There is an implicit assumption that you will either be performing the peers.json procedure on all nodes of a cluster, or only one, whilst wiping and resetting others from scratch:

github.com

hashicorp/raft/blob/6b4e32088e0bda22ea219fc89b0ee47f420e2b0b/api.go#L293-L299


      
          // A typical way to recover the cluster is to shut down all servers and then
          // run RecoverCluster on every server using an identical configuration. When
          // the cluster is then restarted, and election should occur and then Raft will
          // resume normal operation. If it's desired to make a particular server the
          // leader, this can be used to inject a new configuration with that server as
          // the sole voter, and then join up other new clean-state peer servers using
          // the usual APIs in order to bring the cluster back into a known state.

If you are somehow managing to use peers.json only on one node of an existing cluster, then I think that’s an unintended way to use it.