Kubernetes Vault new cluster raft join failing

Vault
, ,
1

Hello,

I’m following the example Kubernetes HA Raft documentation (Highly Available Vault Cluster with Raft | Vault | HashiCorp Developer), but when I try to join additional nodes to the first node, I get the following:

➜  ~ kubectl exec -n vault -ti vault-1 -- vault operator raft join http://vault-0.vault-internal:8200

Error joining the node to the Raft cluster: Error making API request.

URL: POST http://127.0.0.1:8200/v1/sys/storage/raft/join
Code: 500. Errors:

* failed to join raft cluster: failed to get raft challenge
command terminated with exit code 2

I have initialized and unsealed the first node. I confirmed the pods and vault-internal service were named as expected. I notice the pods are not in Ready state but I assume that is because the vault is still sealed.

I have also confirmed that http://vault-0.vault-internal:8200 is reachable from the vault-1 pod using netshoot.

Has the process changed without the documentation catching up?

2

Bump, has anyone else encountered this/have a solution?

It’s extremely frustrating that commands pasted verbatim from the docs fail to work.

3

Is Vault fully unsealed? If I try to join vault-1 before vault-0 is fully unsealed I get the same error. You need to repeat the vault operator unseal command until a quorum of unseal keys is met. By default, vault operator init generates 5 keys with 3 needed for a quorum.

$ kubectl exec -ti vault-1 -- vault operator raft join http://vault-0.vault-internal:8200

Error joining the node to the Raft cluster: Error making API request.
URL: POST http://127.0.0.1:8200/v1/sys/storage/raft/join
Code: 500. Errors:
* failed to join raft cluster: failed to get raft challenge

Verify Vault is fully unsealed and operational on vault-0 before adding other nodes.

$ kubectl exec -ti vault-0 -- vault status

...
Initialized        true
Sealed             false
...