Vault replication between K8S clusters

Vault
1

Usecase: I want to migrate the HA vault setup from a K8S cluster to another K8S cluster.

What I have done:
I have migrated one of the PVCs(data-vault-0) from the source cluster to target cluster and then applied all the objects related to vault on the target cluster with modifying vault statefulset replicas to 1 as I have migrated only one PVC from source. When unsealed with the keys from the source the vault status shows the Active node address as the IP of the pod from the source cluster and the HA mode as standby

Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            1
Threshold               1
Version                 1.16.1
Build Date              2024-04-03T12:35:53Z
Storage Type            raft
Cluster Name            vault-cluster-842346ed
Cluster ID              8761a183-850e-5675-3db2-78ef5efc41ad
HA Enabled              true
HA Cluster              https://vault-0.vault-internal:8201
HA Mode                 standby
Active Node Address     http://10.124.0.5:8200
Raft Committed Index    108
Raft Applied Index      108

When I am trying to login vault with the cluster root token it fails with error

Error authenticating: error looking up token: Get "http://10.124.0.5:8200/v1/auth/token/lookup-self": dial tcp 10.124.0.5:8200: i/o timeout

When I scale the statefulset replicas and try to join vault to the cluster, it errors as follows

kubectl exec vault-1 -n vault -- vault operator raft join http://vault-0.vault-internal:8200
Error joining the node to the Raft cluster: Error making API request.

URL: POST http://127.0.0.1:8200/v1/sys/storage/raft/join
Code: 500. Errors:

* failed to join raft cluster: failed to get raft challenge

The source cluster vault was deployed using helm chart

helm install vault hashicorp/vault --set='ui.enabled=true' --set='ui.serviceType=LoadBalancer' --set='server.ha.enabled=true' --set='server.ha.raft.enabled=true' -n vault --create-namespace

What changes should be done in order to have the replicated vault cluster running?

2

Hi,
I think you should try with snapshot.

Create snapshot on cluster A and restore it on cluster B - do note that since snapshot is from cluster A, recovery keys from cluster A will ALWAYS be needed from initial restore onward, until you rekey. Initial A->B restoring will result in vault sealing and cluster A recovery keys will unlock it.

You would need token with policy that allows to perform snapshot create/restore.
After initial snapshot restore from A to B cluster, subsequent restores on B side should not require unsealing after restoring, since recovery keys are the same.