We have a 3 node raft cluster successfully on k8s. We had to restart all the pods since we had a configmap change. We restarted the followers one by one. The new follower pods came up fine and joined the existing leader. On leader we ran “vault operator step-down” so that a new leader is elected and we could restart the leader pod. However, on running step-down in leader, the followers win the election and for some reason it steps down and this goes on and on with both the followers and the cluster can never elect a leader. Can someone please help us figure put what is going on. Thanks
peers before running step-down
$ vault operator raft list-peers
Node Address State Voter
---- ------- ----- -----
27532da1-7e9b-e0b3-4a37-0e7ceb547b65 127.0.0.1:8201 leader true
78d1025a-ef23-b768-5d07-4fe343225bbe central-vault-2.oci-central-vault.oci.svc.cluster.local:8201 follower true
42d68dd2-c3b1-f679-60d4-df769b26e05a central-vault-1.oci-central-vault.oci.svc.cluster.local:8201 follower true
logs from leader node vault-0
2025-04-24T05:25:50.183Z [WARN] storage.raft: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-24T05:25:50.183Z [INFO] storage.raft: entering candidate state: node="Node at central-vault-0.oci-central-vault.oci.svc.cluster.local:8201 [Candidate]" term=234
2025-04-24T05:25:50.184Z [DEBUG] storage.raft: pre-voting for self: term=234 id=27532da1-7e9b-e0b3-4a37-0e7ceb547b65
2025-04-24T05:25:50.184Z [DEBUG] storage.raft: asking for pre-vote: term=234 from=78d1025a-ef23-b768-5d07-4fe343225bbe address=central-vault-2.oci-central-vault.oci.svc.cluster.local:8201
2025-04-24T05:25:50.184Z [DEBUG] storage.raft: asking for pre-vote: term=234 from=42d68dd2-c3b1-f679-60d4-df769b26e05a address=central-vault-1.oci-central-vault.oci.svc.cluster.local:8201
2025-04-24T05:25:50.184Z [DEBUG] storage.raft: calculated votes needed: needed=2 term=234
2025-04-24T05:25:50.184Z [DEBUG] storage.raft: pre-vote received: from=27532da1-7e9b-e0b3-4a37-0e7ceb547b65 term=234 tally=0
2025-04-24T05:25:50.184Z [DEBUG] storage.raft: pre-vote granted: from=27532da1-7e9b-e0b3-4a37-0e7ceb547b65 term=234 tally=1
2025-04-24T05:25:50.184Z [DEBUG] storage.raft: pre-vote received: from=42d68dd2-c3b1-f679-60d4-df769b26e05a term=234 tally=1
2025-04-24T05:25:50.184Z [DEBUG] storage.raft: pre-vote denied: from=42d68dd2-c3b1-f679-60d4-df769b26e05a term=234 tally=1
2025-04-24T05:25:50.185Z [DEBUG] storage.raft: pre-vote received: from=78d1025a-ef23-b768-5d07-4fe343225bbe term=234 tally=1
2025-04-24T05:25:50.185Z [DEBUG] storage.raft: pre-vote denied: from=78d1025a-ef23-b768-5d07-4fe343225bbe term=234 tally=1
2025-04-24T05:25:50.185Z [INFO] storage.raft: pre-vote campaign failed, waiting for election timeout: term=234 tally=1 refused=2 votesNeeded=2
2025-04-24T05:25:50.254Z [TRACE] core: found new active node information, refreshing
2025-04-24T05:25:52.754Z [TRACE] core: found new active node information, refreshing
2025-04-24T05:25:55.254Z [TRACE] core: found new active node information, refreshing
2025-04-24T05:25:55.662Z [WARN] storage.raft: Election timeout reached, restarting election
2025-04-24T05:25:55.662Z [INFO] storage.raft: entering candidate state: node="Node at central-vault-0.oci-central-vault.oci.svc.cluster.local:8201 [Candidate]" term=234
2025-04-24T05:25:55.662Z [DEBUG] storage.raft: pre-voting for self: term=234 id=27532da1-7e9b-e0b3-4a37-0e7ceb547b65
2025-04-24T05:25:55.662Z [DEBUG] storage.raft: asking for pre-vote: term=234 from=78d1025a-ef23-b768-5d07-4fe343225bbe address=central-vault-2.oci-central-vault.oci.svc.cluster.local:8201
2025-04-24T05:25:55.663Z [DEBUG] storage.raft: asking for pre-vote: term=234 from=42d68dd2-c3b1-f679-60d4-df769b26e05a address=central-vault-1.oci-central-vault.oci.svc.cluster.local:8201
2025-04-24T05:25:55.663Z [DEBUG] storage.raft: calculated votes needed: needed=2 term=234
2025-04-24T05:25:55.663Z [DEBUG] storage.raft: pre-vote received: from=27532da1-7e9b-e0b3-4a37-0e7ceb547b65 term=234 tally=0
2025-04-24T05:25:55.663Z [DEBUG] storage.raft: pre-vote granted: from=27532da1-7e9b-e0b3-4a37-0e7ceb547b65 term=234 tally=1
2025-04-24T05:25:55.663Z [DEBUG] storage.raft: pre-vote received: from=78d1025a-ef23-b768-5d07-4fe343225bbe term=236 tally=1
2025-04-24T05:25:55.663Z [DEBUG] storage.raft: pre-vote denied: found newer term, falling back to follower: term=236
2025-04-24T05:25:55.664Z [INFO] storage.raft: entering follower state: follower="Node at central-vault-0.oci-central-vault.oci.svc.cluster.local:8201 [Follower]" leader-address= leader-id=
2025-04-24T05:25:57.755Z [TRACE] core: found new active node information, refreshing
2025-04-24T05:25:59.685Z [WARN] core.cluster-listener: no TLS config found for ALPN: ALPN=["req_fw_sb-act_v1"]
2025-04-24T05:25:59.685Z [DEBUG] core.cluster-listener: error handshaking cluster connection: error="unsupported protocol"
2025-04-24T05:26:00.256Z [TRACE] core: found new active node information, refreshing
logs from vault -1
2025-04-24T05:26:32.718Z [INFO] storage.raft: entering candidate state: node="Node at central-vault-1.oci-central-vault.oci.svc.cluster.local:8201 [Candidate]" term=243
2025-04-24T05:26:32.718Z [DEBUG] storage.raft: asking for pre-vote: term=243 from=27532da1-7e9b-e0b3-4a37-0e7ceb547b65 address=127.0.0.1:8201
2025-04-24T05:26:32.718Z [DEBUG] storage.raft: asking for pre-vote: term=243 from=78d1025a-ef23-b768-5d07-4fe343225bbe address=central-vault-2.oci-central-vault.oci.svc.cluster.local:8201
2025-04-24T05:26:32.718Z [DEBUG] storage.raft: pre-voting for self: term=243 id=42d68dd2-c3b1-f679-60d4-df769b26e05a
2025-04-24T05:26:32.718Z [DEBUG] storage.raft: calculated votes needed: needed=2 term=243
2025-04-24T05:26:32.718Z [DEBUG] storage.raft: pre-vote received: from=42d68dd2-c3b1-f679-60d4-df769b26e05a term=243 tally=0
2025-04-24T05:26:32.718Z [DEBUG] storage.raft: pre-vote granted: from=42d68dd2-c3b1-f679-60d4-df769b26e05a term=243 tally=1
2025-04-24T05:26:32.718Z [DEBUG] core.cluster-listener: creating rpc dialer: address=127.0.0.1:8201 alpn=raft_storage_v1 host=raft-7577e196-9f1b-8f9d-2ee7-7f7c14b2f5dc
2025-04-24T05:26:32.719Z [DEBUG] storage.raft: pre-vote received: from=78d1025a-ef23-b768-5d07-4fe343225bbe term=243 tally=1
2025-04-24T05:26:32.719Z [DEBUG] storage.raft: pre-vote denied: from=78d1025a-ef23-b768-5d07-4fe343225bbe term=243 tally=1
2025-04-24T05:26:32.720Z [DEBUG] core.cluster-listener: performing server cert lookup
2025-04-24T05:26:32.726Z [DEBUG] core.cluster-listener: performing client cert lookup
2025-04-24T05:26:32.731Z [DEBUG] storage.raft.raft-net: accepted connection: local-address=central-vault-1.oci-central-vault.oci.svc.cluster.local:8201 remote-address=127.0.0.1:43090
2025-04-24T05:26:32.732Z [DEBUG] storage.raft: received a requestPreVote with a newer term, grant the pre-vote
2025-04-24T05:26:32.732Z [DEBUG] storage.raft: pre-vote received: from=27532da1-7e9b-e0b3-4a37-0e7ceb547b65 term=243 tally=1
2025-04-24T05:26:32.732Z [DEBUG] storage.raft: pre-vote granted: from=27532da1-7e9b-e0b3-4a37-0e7ceb547b65 term=243 tally=2
2025-04-24T05:26:32.732Z [INFO] storage.raft: pre-vote successful, starting election: term=243 tally=2 refused=1 votesNeeded=2
2025-04-24T05:26:32.732Z [DEBUG] storage.raft: asking for vote: term=243 from=27532da1-7e9b-e0b3-4a37-0e7ceb547b65 address=127.0.0.1:8201
2025-04-24T05:26:32.732Z [DEBUG] storage.raft: asking for vote: term=243 from=78d1025a-ef23-b768-5d07-4fe343225bbe address=central-vault-2.oci-central-vault.oci.svc.cluster.local:8201
2025-04-24T05:26:32.732Z [DEBUG] storage.raft: voting for self: term=243 id=42d68dd2-c3b1-f679-60d4-df769b26e05a
2025-04-24T05:26:32.733Z [INFO] storage.raft: duplicate requestVote for same term: term=243
2025-04-24T05:26:32.733Z [WARN] storage.raft: duplicate requestVote from: candidate=central-vault-1.oci-central-vault.oci.svc.cluster.local:8201
2025-04-24T05:26:32.733Z [DEBUG] storage.raft: vote granted: from=42d68dd2-c3b1-f679-60d4-df769b26e05a term=243 tally=1
2025-04-24T05:26:32.733Z [DEBUG] storage.raft: vote granted: from=27532da1-7e9b-e0b3-4a37-0e7ceb547b65 term=243 tally=2
2025-04-24T05:26:32.733Z [INFO] storage.raft: election won: term=243 tally=2
2025-04-24T05:26:32.733Z [INFO] storage.raft: entering leader state: leader="Node at central-vault-1.oci-central-vault.oci.svc.cluster.local:8201 [Leader]"
2025-04-24T05:26:32.734Z [INFO] storage.raft: added peer, starting replication: peer=27532da1-7e9b-e0b3-4a37-0e7ceb547b65
2025-04-24T05:26:32.734Z [INFO] storage.raft: added peer, starting replication: peer=78d1025a-ef23-b768-5d07-4fe343225bbe
2025-04-24T05:26:32.734Z [WARN] storage.raft: appendEntries rejected, sending older logs: peer="{Voter 78d1025a-ef23-b768-5d07-4fe343225bbe central-vault-2.oci-central-vault.oci.svc.cluster.local:8201}" next=485
2025-04-24T05:26:32.735Z [INFO] storage.raft: entering follower state: follower="Node at central-vault-1.oci-central-vault.oci.svc.cluster.local:8201 [Follower]" leader-address= leader-id=
2025-04-24T05:26:32.735Z [WARN] storage.raft: appendEntries rejected, sending older logs: peer="{Voter 78d1025a-ef23-b768-5d07-4fe343225bbe central-vault-2.oci-central-vault.oci.svc.cluster.local:8201}" next=484
2025-04-24T05:26:35.987Z [DEBUG] core: forwarding: error sending echo request to active node: error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: remote error: tls: internal error\""
logs from vault-2
2025-04-24T05:27:25.751Z [ERROR] core: failed to acquire lock: error="node is not the leader"
2025-04-24T05:27:26.695Z [WARN] storage.raft: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2025-04-24T05:27:26.695Z [INFO] storage.raft: entering candidate state: node="Node at central-vault-2.oci-central-vault.oci.svc.cluster.local:8201 [Candidate]" term=253
2025-04-24T05:27:26.695Z [DEBUG] storage.raft: asking for pre-vote: term=253 from=27532da1-7e9b-e0b3-4a37-0e7ceb547b65 address=127.0.0.1:8201
2025-04-24T05:27:26.695Z [DEBUG] storage.raft: pre-voting for self: term=253 id=78d1025a-ef23-b768-5d07-4fe343225bbe
2025-04-24T05:27:26.695Z [DEBUG] storage.raft: asking for pre-vote: term=253 from=42d68dd2-c3b1-f679-60d4-df769b26e05a address=central-vault-1.oci-central-vault.oci.svc.cluster.local:8201
2025-04-24T05:27:26.695Z [DEBUG] storage.raft: calculated votes needed: needed=2 term=253
2025-04-24T05:27:26.696Z [DEBUG] storage.raft: pre-vote received: from=78d1025a-ef23-b768-5d07-4fe343225bbe term=253 tally=0
2025-04-24T05:27:26.696Z [DEBUG] storage.raft: pre-vote granted: from=78d1025a-ef23-b768-5d07-4fe343225bbe term=253 tally=1
2025-04-24T05:27:26.696Z [DEBUG] storage.raft: received a requestPreVote with a newer term, grant the pre-vote
2025-04-24T05:27:26.696Z [DEBUG] storage.raft: pre-vote received: from=27532da1-7e9b-e0b3-4a37-0e7ceb547b65 term=253 tally=1
2025-04-24T05:27:26.696Z [DEBUG] storage.raft: pre-vote granted: from=27532da1-7e9b-e0b3-4a37-0e7ceb547b65 term=253 tally=2
2025-04-24T05:27:26.696Z [INFO] storage.raft: pre-vote successful, starting election: term=253 tally=2 refused=0 votesNeeded=2
2025-04-24T05:27:26.697Z [DEBUG] storage.raft: asking for vote: term=253 from=27532da1-7e9b-e0b3-4a37-0e7ceb547b65 address=127.0.0.1:8201
2025-04-24T05:27:26.697Z [DEBUG] storage.raft: voting for self: term=253 id=78d1025a-ef23-b768-5d07-4fe343225bbe
2025-04-24T05:27:26.698Z [DEBUG] storage.raft: asking for vote: term=253 from=42d68dd2-c3b1-f679-60d4-df769b26e05a address=central-vault-1.oci-central-vault.oci.svc.cluster.local:8201
2025-04-24T05:27:26.698Z [INFO] storage.raft: duplicate requestVote for same term: term=253
2025-04-24T05:27:26.698Z [WARN] storage.raft: duplicate requestVote from: candidate=central-vault-2.oci-central-vault.oci.svc.cluster.local:8201
2025-04-24T05:27:26.698Z [DEBUG] storage.raft: vote granted: from=78d1025a-ef23-b768-5d07-4fe343225bbe term=253 tally=1
2025-04-24T05:27:26.698Z [DEBUG] storage.raft: vote granted: from=27532da1-7e9b-e0b3-4a37-0e7ceb547b65 term=253 tally=2
2025-04-24T05:27:26.698Z [INFO] storage.raft: election won: term=253 tally=2
2025-04-24T05:27:26.698Z [INFO] storage.raft: entering leader state: leader="Node at central-vault-2.oci-central-vault.oci.svc.cluster.local:8201 [Leader]"
2025-04-24T05:27:26.698Z [INFO] storage.raft: added peer, starting replication: peer=27532da1-7e9b-e0b3-4a37-0e7ceb547b65
2025-04-24T05:27:26.698Z [INFO] storage.raft: added peer, starting replication: peer=42d68dd2-c3b1-f679-60d4-df769b26e05a
2025-04-24T05:27:26.700Z [INFO] storage.raft: entering follower state: follower="Node at central-vault-2.oci-central-vault.oci.svc.cluster.local:8201 [Follower]" leader-address= leader-id=
2025-04-24T05:27:26.701Z [WARN] storage.raft: appendEntries rejected, sending older logs: peer="{Voter 42d68dd2-c3b1-f679-60d4-df769b26e05a central-vault-1.oci-central-vault.oci.svc.cluster.local:8201}" next=490
2025-04-24T05:27:30.657Z [DEBUG] core: forwarding: error sending echo request to active node: error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: remote error: tls: internal error\""
2025-04-24T05:27:30.880Z [WARN] storage.raft: rejecting pre-vote request since our last term is greater: candidate=central-vault-0.oci-central-vault.oci.svc.cluster.local:8201 last-term=253 last-candidate-term=3
vault config below. Pod name placeholder gets replaced before starting vault server.
vault_config.json: |
{
"default_lease_ttl": "600s",
"max_lease_ttl": "600s",
"disable_mlock": true,
"api_addr": "https://{POD_NAME_PLACEHOLDER}.oci-central-vault.oci.svc.cluster.local:8200",
"cluster_addr": "https://{POD_NAME_PLACEHOLDER}.oci-central-vault.oci.svc.cluster.local:8201",
"listener": {
"tcp": {
"address": "0.0.0.0:8200",
"cluster_address": "0.0.0.0:8201",
"tls_cert_file": "/conf/tls.crt",
"tls_key_file": "/conf/tls.key"
}
},
"storage": {
"raft": {
"path": "/datadir/central-vault/{POD_NAME_PLACEHOLDER}",
"retry_join":[
{
"leader_api_addr": "https://central-vault-0.oci-central-vault.oci.svc.cluster.local:8200",
"leader_ca_cert_file": "/conf/ca.crt",
"leader_client_cert_file": "/conf/tls.crt",
"leader_client_key_file": "/conf/tls.key"
},
{
"leader_api_addr": "https://central-vault-1.oci-central-vault.oci.svc.cluster.local:8200",
"leader_ca_cert_file": "/conf/ca.crt",
"leader_client_cert_file": "/conf/tls.crt",
"leader_client_key_file": "/conf/tls.key"
},
{
"leader_api_addr": "https://central-vault-2.oci-central-vault.oci.svc.cluster.local:8200",
"leader_ca_cert_file": "/conf/ca.crt",
"leader_client_cert_file": "/conf/tls.crt",
"leader_client_key_file": "/conf/tls.key"
}
]
}
}