I have initialized the vault Cluster using AWSKMS.
It has given me the recovery keys and the Root Token.
I need to unseal the other nodes with the UNSEAL KEY which is in AWS KMS.
How do I unseal the other nodes. Only the first node is unsealed with "vault operator init.
Poor Documentation around this topic. Expansive examples about how the AUTO UNSEAL works in a HA Cluster would be tremendously helpful.
The easiest option would be to just restart them, and let them undergo auto-unseal automatically as they start.
You could also command them to make a fresh auto-unseal attempt without a restart, for which you’d need a quorum of recovery keys as authentication, but restarting is probably easier?
No, I don’t think I can do this as the nodes haven’t joined the Cluster yet.
The Unseal part is not done yet. (Which uses the Cluster URL to join the Vault Cluster).
What is the command to auto-unseal the other nodes, I do not have the UNSEAL KEY, as it is in KMS.
The command I know is,
kubectl exec -n $NAMESPACE $VAULT_RELEASE_NAME-0 – vault operator unseal -address=https://$VAULT_RELEASE_NAME-0.$VAULT_RELEASE_NAME-internal:8200 $VAULT_UNSEAL_KEY.
I do not have the UNSEAL KEY here
Is this a Raft cluster? You have not mentioned the storage backend in use.
If so, you have the order of operations backwards - you must first join the cluster. At the moment of joining the cluster, the additional nodes will use the auto-unseal configuration to automatically unseal, without further action from you.
UNSEAL : AWS KMS
CLUSTER : RAFT PROTOCOL
HA ENABLED: Yes
No, I cannot join a Cluster which is not Unsealed.
First, I unsealed one node, I got the Recovery Keys and the Root Token.
Next, I joined the other nodes to this Cluster.
They still remain Sealed.
For joining a Cluster, I used this command :
kubectl exec -n $NAMESPACE $VAULT_RELEASE_NAME-$i -- vault operator raft join -address=https://$VAULT_RELEASE_NAME-$i.$VAULT_RELEASE_NAME-internal:8200 -leader-ca-cert="$(cat vault.ca)" -leader-client-cert="$(cat vault.crt)" -leader-client-key="$(cat vault.key)" https://$VAULT_RELEASE_NAME-0.$VAULT_RELEASE_NAME-internal:8200
The join operation requires the URL, which works only when the Cluster has atleast one node unsealed.
With a new Cluster, I tried running the raft join command,
this is what I see, I’ve used the same command as stated above, to join.
Error joining the node to the Raft cluster: Error making API request.
URL: POST https://RELEASE-NAME-0.$RELEASE-NAME-internal:8200/v1/sys/storage/raft/join
Code: 500. Errors:
My Helm Values file has the necessary ENV VARS set for AWS KMS. (
In server-> extraEnvironmentVars, I’ve set,
VAULT_SEAL_TYPE: awskms
AWS_REGION: redacted
AWS_ACCESS_KEY_ID: redacted
AWS_SECRET_ACCESS_KEY: redacted
VAULT_AWSKMS_SEAL_KEY_ID: redacted)
TLS is enabled
That is true - but what I was pointing out was that the node that is joining, does not need to be unsealed before joining - and in fact cannot be.
You mean initialized, I guess, from the context.
And here something has gone wrong. When you perform a successful join, in a cluster configured with auto-unseal, the newly joined node unseals itself.
(I confirmed this with a quick local test using transit auto-unseal.)
You should review the server logs looking for something going wrong.
There’s nothing that I can see wrong with this, but you might want to consider specifying a retry_join block in the Vault configuration file, which will allow you to skip needing to manually manage when to run the command.
Nice, thanks @maxb for helping and all of your replies were quick too, so thanks again.
The retry_join did the trick, I just unsealed the first one and the rest joined.