Vault setup on kubernetes using operator & migrating data from one vault to another

Hi Team,

My requirement is - Setting up highly available vault on kubernetes cluster & Migrating data from existing vault to new vault(which will be setup on k8s).

Couple of questions on this

  • Do we have any operator which is recommended by Hashicorp for setting up vault on kubernetes cluster?
  • what is the best way for migrating secrets from one vault to another?

This depends partly on your setup. So perhaps answering some questions first would help.

  1. Do you store your unseal keys? And how?
  2. What kind of storage do you use?

I am planning to leverage autounseal concept i.e “auto unseal with aws kms” and etcd as storage backend.

Thank you for the information! You mention “planning to”, what are you currently using?

details are as follows.

Existing Vault:

aws kms for unseal keys
etcd as storage backend

New Vault(yet to create in k8s)

aws kms for unseal keys
Integrated storage as backend

WIth that setup there are two best possibilities, one easier than the other.

  1. You migrate (using vault operator migrate) from etcd to Raft storage.
    After this you can follow the standard backup guide to then restore the data on the new cluster.
    AWS KMS will take care of the rest of the process.
  2. Backup the etcd cluster only to then restore the etcd cluster on the new cluster.
    AWS KMS will take care of the rest of the process.

Some notes:

  • I recommend if you decide to get with the first option to backup Raft on your Raft leader. You can identify the leader in the outcome of the command vault operator raft list-peers
  • When migrating, due to that you export the data outside of Vault, I recommend rotating your encryption keys.

Thanks a lot for the info. I will give a try on the first option.

hi can you please help me on migration of below scenario.
old setup
gcp kms for unseal keys
gcp storage bucket for backend

new setup
was kms for unseal
raft for backend

I have managed the unseal part but badly stuck with backend

The same steps would work for you. Use the vault operator migrate command to migrate from GCP storage to Raft.

Thanks for your reply I have tried the same steps but getting below error, the issue may be because vault is running mode but not sure how can I stop in pod

Error migrating: error mounting ‘storage_destination’: failed to create fsm: failed to open bolt file: timeout

What is your mount setup? Permissions, mount type, etc. And what kind of cluster setup do you have? How are they connected?

here is my values.yaml file

global:
enabled: true
tlsDisable: false
injector:
enabled: true
replicas: 1
port: 8080
leaderElector:
enabled: true
metrics:
enabled: false
injector.
image:
repository: “hashicorp/vault-k8s”
tag: “0.14.2”
pullPolicy: IfNotPresent
agentImage:
repository: “hashicorp/vault”
tag: “1.9.3”
cpuLimit: “500m”
cpuRequest: “250m”
memLimit: “128Mi”
memRequest: “64Mi”
exitOnRetryFailure: true
staticSecretRenderInterval: “”
logLevel: “info”
logFormat: “standard”
revokeOnShutdown: false
affinity: |
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app.kubernetes.io/name: {{ template “vault.name” . }}-agent-injector
app.kubernetes.io/instance: “{{ .Release.Name }}”
component: webhook
topologyKey: kubernetes.io/hostname

server:
enabled: true.
secretKey: “license”
image:
repository: “hashicorp/vault”
tag: “1.9.3”
failureThreshold: 2
initialDelaySeconds: 5
successThreshold: 1
timeoutSeconds: 3
livenessProbe:
enabled: false
path: “/v1/sys/health?standbyok=true”
failureThreshold: 2
initialDelaySeconds: 60
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 3
terminationGracePeriodSeconds: 10
extraVolumes:
- type: secret
name: vault-tls
affinity: |
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app.kubernetes.io/name: {{ template “vault.name” . }}
app.kubernetes.io/instance: “{{ .Release.Name }}”
component: server
topologyKey: kubernetes.io/hostname

service:
enabled: true
dataStorage:
enabled: true
size: 50Gi
mountPath: “/vault/data”
storageClass: null
enabled: false
size: 50Gi
mountPath: “/vault/audit”
ha:
enabled: true
replicas: 3
apiAddr: null
raft:
enabled: true
setNodeId: true
config: |
ui = true

    listener "tcp" {

    address = "[::]:8200"
    cluster_address = "[::]:8201"
    tls_cert_file = "/vault/userconfig/vault-tls/vault.crt"
    tls_key_file = "/vault/userconfig/vault-tls/vault.key"
    }
    storage "raft" {
      path = "/vault/data"
    }

    service_registration "kubernetes" {}
    
    seal "awskms" {
      kms_key_id = “xxxxxx”
    disruptionBudget:
  enabled: true
  maxUnavailable: null

serviceAccount:
create: true
name: “vault-sa”
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxx:role/vault

it is mentioned in the documentation that vault should be offline during migration but I am running in pod so not sure how can I shutdown in pod

I am missing the entry for GCP storage. For Vault to be able to read out the GCP storage it first needs to read it out. Let’s start with that! :slightly_smiling_face:

I have migrated all the data to s3 bucket then then I logged in to my pod and created migrate.hcl file with below content
storage_source “s3” {
bucket = “xxxxx”
region = “xxxxx”
access_key = “xxxxx”
secret_key = “xxxxx”
session_token = “”

}

storage_destination “raft” {
path = “/vault/data”
node_id = “vault-0”
}
cluster_addr = “http://vault-0.vault-internal:8201

but when I am running the migrator operator getting the below error

Error migrating: error mounting ‘storage_destination’: failed to create fsm: failed to open bolt file: timeout

If possible can you look into this Using Letsencrypt certificates for Vault setup on kubernetes query

I have migrated the data from gcp bucket to s3 bucket that’s why my source is s3 also if you can see the issue is with destination not the source, I have gone through it’s document operator migrate - Command | Vault by HashiCorp they mentioned that it’s an offline but I am doing it in k8 cluster so how can I stop vault

@purushothamkdr143 have you managed to migrate your data?