Trouble unsealing vault: raft ha instance

hey, I’m having weird troubles unsealing my vault ha setup with raft. i have 3 separate vm where i installed vault 1.14.9 on them.

this is my vault.hcl:

ui = true

disable_mlock = true

storage “raft” {
retry_join {
leader_api_addr = “https://node2:8200
leader_ca_cert_file = “/opt/vault/tls/vault-ca.pem”
leader_client_cert_file = “/opt/vault/tls/vault-cert.pem”
leader_client_key_file = “/opt/vault/tls/vault-key.pem”
}
retry_join {
leader_api_addr = “https://node1:8200
leader_ca_cert_file = “/opt/vault/tls/vault-ca.pem”
leader_client_cert_file = “/opt/vault/tls/vault-cert.pem”
leader_client_key_file = “/opt/vault/tls/vault-key.pem”
}
retry_join {
leader_api_addr = “https://node3:8200
leader_ca_cert_file = “/opt/vault/tls/vault-ca.pem”
leader_client_cert_file = “/opt/vault/tls/vault-cert.pem”
leader_client_key_file = “/opt/vault/tls/vault-key.pem”
}
path = “/opt/vault/data”
node_id = “vault-02”
}

listener “tcp” {
address = “0.0.0.0:8200”
tls_cert_file = “/opt/vault/tls/vault-cert.pem”
tls_key_file = “/opt/vault/tls/vault-key.pem”
tls_client_ca_file = “/opt/vault/tls/vault-ca.pem”
}

cluster_addr = “https://node2:8201
api_addr = “https://loadbalancer.url.com:8200

(this is the hcl of vault-02)

as you can see, this is ansible managed. the only thing that is changing across the three nodes is the node_id, vault-01,vault-02 or vault-03 with vault-01 being the leader, and the cluster_addr is always the node its running on.

unsealing vault works fine, when i unseal 02 or 03, it seems to work fine as well, i enter the 3 unseal keys generated from init on vault-01. I get no error message or similar. when i check

vault operator raft list-peers
Node Address State Voter


vault-01 node01:8201 leader true
vault-02 node02:8201 follower false

vault-02 seems to have joined.

but vault-02 is still in a sealed state. when i retry to unseal i get the following message:

Error unsealing: Error making API request.

URL: PUT https://127.0.0.1:8200/v1/sys/unseal
Code: 500. Errors:

  • failed to create cipher: crypto/aes: invalid key size 0

i also tried restarting the service and retrying it, but then the unseal process just times out.

this is a maybe relevant block from the logs:

Feb 12 10:05:07 vault-02 vault[19516]: error=
Feb 12 10:05:07 vault-02 vault[19516]: | error during raft bootstrap init call: Error making API request.
Feb 12 10:05:07 vault-02 vault[19516]: |
Feb 12 10:05:07 vault-02 vault[19516]: | URL: PUT https://node1:8200/v1/sys/storage/raft/bootstrap/challenge
Feb 12 10:05:07 vault-02 vault[19516]: | Code: 503. Errors:
Feb 12 10:05:07 vault-02 vault[19516]: |
Feb 12 10:05:07 vault-02 vault[19516]: | * Vault is sealed
Feb 12 10:05:07 vault-02 vault[19516]:
Feb 12 10:05:07 vault-02 vault[19516]: 2024-02-12T10:05:07.483+0100 [ERROR] core: failed to retry join raft cluster: retry=2s
Feb 12 10:05:07 vault-02 vault[19516]: err=
Feb 12 10:05:07 vault-02 vault[19516]: | failed to send answer to raft leader node: Error making API request.
Feb 12 10:05:07 vault-02 vault[19516]: |
Feb 12 10:05:07 vault-02 vault[19516]: | URL: PUT https://node3:8200/v1/sys/storage/raft/bootstrap/answer
Feb 12 10:05:07 vault-02 vault[19516]: | Code: 500. Errors:
Feb 12 10:05:07 vault-02 vault[19516]: |
Feb 12 10:05:07 vault-02 vault[19516]: | * Preventing server addition that would require removal of too many servers and cause cluster instability

okay i figured out the problem: i forgot to open up the tcp port 8201 in our firewall.