I manage to simulate this scenario blocking traffic from the current leader likes described here: 3-node cluster unhealthy after leader lost network connection - #2 by Ranjandas, using:
# block inbound RPC
iptables -I INPUT -p tcp --dport 8300 -j DROP
# block inbound Serf LAN & WAN
iptables -I INPUT -p tcp --dport 8301 -j DROP
iptables -I INPUT -p tcp --dport 8301 -j DROP
iptables -I INPUT -p tcp --dport 8302 -j DROP
iptables -I INPUT -p udp --dport 8302 -j DROP
# block outbound RPC
iptables -I OUTPUT -p tcp --dport 8300 -j DROP
# block outbound Serf LAN & WAN
iptables -I OUTPUT -p tcp --dport 8301 -j DROP
iptables -I OUTPUT -p udp --dport 8301 -j DROP
iptables -I OUTPUT -p tcp --dport 8302 -j DROP
iptables -I OUTPUT -p udp --dport 8302 -j DROP
This is a test cluster with no ACL and when I run consul operator raft list-peers, I get this error message:
Error getting peers: Failed to retrieve raft configuration: Unexpected response code: 403 (rpc error making call: ACL not found)
How could I recover the cluster or promote a new leader?
This is the output of consul operator raft list-peers -stale
Node ID Address State Voter RaftProtocol
eu-consul-2 b1c0dcc1-ed95-9915-702e-86c85adec93e 188.34.185.115:8300 follower true 3
us-consul-2 40f7426b-96ff-daea-8b82-fc4e96615fd3 5.161.151.80:8300 leader true 3
eu-consul-3 befc558f-16c9-3a58-768b-449730eeac24 49.12.7.233:8300 follower true 3
eu-consul-1 6ce38127-5bfa-744b-8ca7-88baf1c5cc23 78.46.187.173:8300 follower true 3