Raft cluster status shows node up and running despite being down

Hello,

I would like to understand how vault knows if there are issues with a node that is no longer reacher or it’s shut down.
I’ve shut down vault-2, but the status isn’t being updated when I list the peers.

root@vault-3-staging:~# vault operator raft list-peers
Node                                    Address                                     State       Voter
----                                    -------                                     -----       -----
9ca56370-f00f-f327-85de-96aed1f88c36    vault-1-staging.node.company.internal:8201    follower    true
fd303e8d-4b10-3814-0e0a-11c8b2ce7964    vault-2-staging.node.company.internal:8201    follower    true
92bb2cd1-f3c7-d549-4b47-121abbf4b33c    vault-3-staging.node.company.internal:8201    leader      true

I’m not sure what to expect exactly, maybe this isn’t where I’m supposed to find this information and when I look into the vault-3’s logs, I do see that it cannot reach vault-2:

Aug 04 13:09:22 vault-3-staging vault[3120]: 2022-08-04T13:09:22.064+0300 [ERROR] storage.raft: failed to heartbeat to: peer=vault-2-staging.node.company.internal:8201 error="dial tcp 10.16.16.151:8201: connect: connection refused"

So my question is, how do I know using if something if a node is unreachable using vault’s tools directly (command line etc.)?

See cluster status

1 Like