I’m deploying a new 3 node vault cluster on kubernetes using helm charts. I’m using the hashicorp vault-helm charts at hashicorp/vault-helm: Helm chart to install Vault and other associated components.
The pertinent parts:
raft:
config: |
ui = true
listener "tcp" {
tls_disable = 0
address = "[::]:8200"
cluster_address = "[::]:8201"
tls_cert_file = "/vault/userconfig/vault-ha-tls/vault.crt"
tls_key_file = "/vault/userconfig/vault-ha-tls/vault.key"
tls_client_ca_file = "/vault/userconfig/vault-ha-tls/vault.ca"
# Enable unauthenticated metrics access (necessary for Prometheus Operator)
#telemetry {
# unauthenticated_metrics_access = "true"
#}
}
storage "raft" {
path = "/vault/data"
retry_join {
leader_api_addr = "https://vault-0.vault-internal:8200"
leader_ca_cert_file = "/vault/userconfig/vault-ha-tls/vault.ca"
leader_client_cert_file = "/vault/userconfig/vault-ha-tls/vault.crt"
leader_client_key_file = "/vault/userconfig/vault-ha-tls/vault.key"
}
retry_join {
leader_api_addr = "https://vault-1.vault-internal:8200"
leader_ca_cert_file = "/vault/userconfig/vault-ha-tls/vault.ca"
leader_client_cert_file = "/vault/userconfig/vault-ha-tls/vault.crt"
leader_client_key_file = "/vault/userconfig/vault-ha-tls/vault.key"
}
retry_join {
leader_api_addr = "https://vault-2.vault-internal:8200"
leader_ca_cert_file = "/vault/userconfig/vault-ha-tls/vault.ca"
leader_client_cert_file = "/vault/userconfig/vault-ha-tls/vault.crt"
leader_client_key_file = "/vault/userconfig/vault-ha-tls/vault.key"
}
}
disable_mlock = true
service_registration "kubernetes" {}
enabled: true
setNodeId: true
replicas: 3
My understanding from documentation is that the nodes should automatically join the leader node that is initialized and unsealed. The raft comes to consensus and that I should be able to use the root token/unseal keys from leader on the other nodes. In my cluster, I initialized and unsealed vault-2. I can see from list-peers that vault-0 has joined but is not a voter(suggests a problem here).
kubectl exec -it vault-2 -- /bin/sh
/ $ vault status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.19.0
Build Date 2025-03-04T12:36:40Z
Storage Type raft
Cluster Name vault-cluster-d06fa999
Cluster ID 4d7ee770-5ad5-806d-4bf8-d593603e6748
Removed From Cluster false
HA Enabled true
HA Cluster https://vault-2.vault-internal:8201
HA Mode active
Active Since 2025-05-23T15:43:48.072301656Z
Raft Committed Index 50
Raft Applied Index 50
/ $ vault operator raft list-peers
Node Address State Voter
---- ------- ----- -----
vault-2 vault-2.vault-internal:8201 leader true
vault-0 vault-0.vault-internal:8201 follower false
/ $
I’ve issued no other commands in the vault cluster. I’ve only interacted with vault-2 at this point. However, I can see on the vault-0 node that it has been initialized. I did not do this init. I have no unseal keys for vault-0 node.
kubectl exec -it vault-0 -- /bin/sh
/ $ vault status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 0/3
Unseal Nonce n/a
Version 1.19.0
Build Date 2025-03-04T12:36:40Z
Storage Type raft
Removed From Cluster false
HA Enabled true
As reported by vault status, the node is sealed. I am not able to unseal it using the keys from vault-2. I don’t believe this is expected behavior.
What am I missing?