We have a 3 node vault cluster which has been logging warning “core.raft: skipping new raft TLS config creation, keys are pending” for 4-5 days now.
vault status command on all three nodes shows both “Raft Applied Index” and “Raft Committed Index” to be in sync with each nodes.
What would be the best way to mitigate this?
Which ports are open between the nodes?
The required ports 8200 and 8201 are open. I see that nodes are able to communicate with each other. No other error logs w.r.t vault nodes communicating with each other.
vault operator raft list-peers also gives correct details about all three nodes.
Node Address State Voter
---- ------- ----- -----
node_0 x.x.x.x:8201 leader true
node_1 x.x.x.x:8201 follower true
node_2 x.x.x.x:8201 follower true
I am facing the same situation. Three nodes cluster and getting the subject error on the cluster leader repeatedly after 5 minutes.
Vault status output:
$ vault status
Key Value
--- -----
Recovery Seal Type shamir
Initialized true
Sealed false
Total Recovery Shares 5
Threshold 3
Version 1.12.2
Build Date 2022-11-23T12:53:46Z
Storage Type raft
Cluster Name vault-cluster-30f98569
Cluster ID 6912b601-5d8a-a1af-bf30-4c04333ccc6c
HA Enabled true
HA Cluster https://host.name.com:8201
HA Mode active
Active Since 2022-12-29T08:45:14.775477414Z
Raft Committed Index 2043
Raft Applied Index 2043
Vault raft peers status:
$ vault operator raft list-peers
Node Address State Voter
---- ------- ----- -----
vault1 172.16.X.X:8201 leader true
vault2 172.16.X.X:8201 follower true
vault3 172.16.X.X:8201 follower true
Any help would be appreciated on how to address this.
Stopping and starting the raft leader has apparently fixed the issue.
Jan 05 13:27:33 vault.server.com vault[70101]: 2023-01-05T13:27:33.548+0500 [INFO] rollback: starting rollback manager
Jan 05 13:27:33 vault.server.com vault[70101]: 2023-01-05T13:27:33.549+0500 [INFO] core: restoring leases
Jan 05 13:27:33 vault.server.com vault[70101]: 2023-01-05T13:27:33.557+0500 [INFO] identity: entities restored
Jan 05 13:27:33 vault.server.com vault[70101]: 2023-01-05T13:27:33.557+0500 [INFO] identity: groups restored
Jan 05 13:27:33 vault.server.com vault[70101]: 2023-01-05T13:27:33.557+0500 [INFO] core: starting raft active node
Jan 05 13:27:33 vault.server.com vault[70101]: 2023-01-05T13:27:33.558+0500 [INFO] storage.raft: starting autopilot: config="&{false 0 10s 24h0m0s 1000 0 10s false redundancy_zone upgrade_version}" reconci>
Jan 05 13:27:33 vault.server.com vault[70101]: 2023-01-05T13:27:33.560+0500 [INFO] expiration: lease restore complete
Jan 05 13:27:33 vault.server.com vault[70101]: 2023-01-05T13:27:33.561+0500 [INFO] core: usage gauge collection is disabled
Jan 05 13:27:34 vault.server.com vault[70101]: 2023-01-05T13:27:34.430+0500 [INFO] core: post-unseal setup complete
Jan 05 13:27:35 vault.server.com vault[70101]: 2023-01-05T13:27:35.533+0500 [INFO] secrets.database.database_23bb7180: populating role rotation queue
Jan 05 13:27:35 vault.server.com vault[70101]: 2023-01-05T13:27:35.677+0500 [INFO] secrets.database.database_23bb7180: starting periodic ticker
Jan 05 13:28:33 vault.server.com vault[70101]: 2023-01-05T13:28:33.559+0500 [WARN] core.raft: skipping new raft TLS config creation, keys are pending
Jan 05 13:28:33 vault.server.com vault[70101]: 2023-01-05T13:28:33.936+0500 [INFO] core.raft: installed new raft TLS key: term=22
Jan 05 13:33:43 vault.server.com vault[70101]: 2023-01-05T13:33:43.560+0500 [INFO] core.raft: creating new raft TLS config
Jan 05 13:33:45 vault.server.com vault[70101]: 2023-01-05T13:33:45.728+0500 [INFO] core.raft: wrote new raft TLS config
Jan 05 13:34:33 vault.server.com vault[70101]: 2023-01-05T13:34:33.865+0500 [INFO] core.raft: installed new raft TLS key: term=24