hey, I’m having weird troubles unsealing my vault ha setup with raft. i have 3 separate vm where i installed vault 1.14.9 on them.
this is my vault.hcl:
ui = true
disable_mlock = true
storage “raft” {
retry_join {
leader_api_addr = “https://node2:8200”
leader_ca_cert_file = “/opt/vault/tls/vault-ca.pem”
leader_client_cert_file = “/opt/vault/tls/vault-cert.pem”
leader_client_key_file = “/opt/vault/tls/vault-key.pem”
}
retry_join {
leader_api_addr = “https://node1:8200”
leader_ca_cert_file = “/opt/vault/tls/vault-ca.pem”
leader_client_cert_file = “/opt/vault/tls/vault-cert.pem”
leader_client_key_file = “/opt/vault/tls/vault-key.pem”
}
retry_join {
leader_api_addr = “https://node3:8200”
leader_ca_cert_file = “/opt/vault/tls/vault-ca.pem”
leader_client_cert_file = “/opt/vault/tls/vault-cert.pem”
leader_client_key_file = “/opt/vault/tls/vault-key.pem”
}
path = “/opt/vault/data”
node_id = “vault-02”
}listener “tcp” {
address = “0.0.0.0:8200”
tls_cert_file = “/opt/vault/tls/vault-cert.pem”
tls_key_file = “/opt/vault/tls/vault-key.pem”
tls_client_ca_file = “/opt/vault/tls/vault-ca.pem”
}cluster_addr = “https://node2:8201”
api_addr = “https://loadbalancer.url.com:8200”
(this is the hcl of vault-02)
as you can see, this is ansible managed. the only thing that is changing across the three nodes is the node_id, vault-01,vault-02 or vault-03 with vault-01 being the leader, and the cluster_addr is always the node its running on.
unsealing vault works fine, when i unseal 02 or 03, it seems to work fine as well, i enter the 3 unseal keys generated from init on vault-01. I get no error message or similar. when i check
vault operator raft list-peers
Node Address State Voter
vault-01 node01:8201 leader true
vault-02 node02:8201 follower false
vault-02 seems to have joined.
but vault-02 is still in a sealed state. when i retry to unseal i get the following message:
Error unsealing: Error making API request.
URL: PUT https://127.0.0.1:8200/v1/sys/unseal
Code: 500. Errors:
- failed to create cipher: crypto/aes: invalid key size 0
i also tried restarting the service and retrying it, but then the unseal process just times out.
this is a maybe relevant block from the logs:
Feb 12 10:05:07 vault-02 vault[19516]: error=
Feb 12 10:05:07 vault-02 vault[19516]: | error during raft bootstrap init call: Error making API request.
Feb 12 10:05:07 vault-02 vault[19516]: |
Feb 12 10:05:07 vault-02 vault[19516]: | URL: PUT https://node1:8200/v1/sys/storage/raft/bootstrap/challenge
Feb 12 10:05:07 vault-02 vault[19516]: | Code: 503. Errors:
Feb 12 10:05:07 vault-02 vault[19516]: |
Feb 12 10:05:07 vault-02 vault[19516]: | * Vault is sealed
Feb 12 10:05:07 vault-02 vault[19516]:
Feb 12 10:05:07 vault-02 vault[19516]: 2024-02-12T10:05:07.483+0100 [ERROR] core: failed to retry join raft cluster: retry=2s
Feb 12 10:05:07 vault-02 vault[19516]: err=
Feb 12 10:05:07 vault-02 vault[19516]: | failed to send answer to raft leader node: Error making API request.
Feb 12 10:05:07 vault-02 vault[19516]: |
Feb 12 10:05:07 vault-02 vault[19516]: | URL: PUT https://node3:8200/v1/sys/storage/raft/bootstrap/answer
Feb 12 10:05:07 vault-02 vault[19516]: | Code: 500. Errors:
Feb 12 10:05:07 vault-02 vault[19516]: |
Feb 12 10:05:07 vault-02 vault[19516]: | * Preventing server addition that would require removal of too many servers and cause cluster instability
