Consul server not a voter

normally it takes a bit before a server becomes a valid voter, but I have this server that just doesn’t want to be “voter = true”

how do you troubleshoot something like this?

Hi @dilaodilao,

What version of Consul are you running? Is it Enterprise or Community Edition? Also, are all the server agents running the exact same version?

I would recommend checking the logs on the leader and the affected node to see if there are any reasons logged.

In addition, check the output of consul operator autopilot state command. Check the Health Status of the node.

If you are running the enterprise edition, there is a feature called Upgrade Migration that could cause a similar behaviour. ref: Automate upgrades with Consul Enterprise | Consul | HashiCorp Developer

community edition, 1.13.1

running the autopilot state lists
status and node type as voter
node status is alive
nothing unordinary, compared to the other voting server nodes.

while I dig at the logs, is this circumstance possible?
where you have a 5 node cluster, (which allows 2 failures), can you be in a state where 2 of them (for whatever reason, can not vote) and you lose 1 node
would that cause the entire cluster to fail (no leader selection)?

Yes, the scenario you described is possible. Because in a 5-node cluster, if you have 2 non-voters, then you are left with 3 nodes that have voting rights. In such case, if you lose one voter, you lose the quorum and the cluster will fail to elect a leader.

If you are in a situation where out of five nodes, 2 are non-voter for whatever reason, it is better to make them leave the cluster using consul leave. This would bring down your cluster size to 3 safely, and then even if you lose another healthy node, the cluster would survive with 2 nodes (as the quorum is 2).