Vault setup on kubernetes using operator & migrating data from one vault to another

Vault
1

Hi Team,

My requirement is - Setting up highly available vault on kubernetes cluster & Migrating data from existing vault to new vault(which will be setup on k8s).

Couple of questions on this

  • Do we have any operator which is recommended by Hashicorp for setting up vault on kubernetes cluster?
  • what is the best way for migrating secrets from one vault to another?
2

This depends partly on your setup. So perhaps answering some questions first would help.

  1. Do you store your unseal keys? And how?
  2. What kind of storage do you use?
3

I am planning to leverage autounseal concept i.e “auto unseal with aws kms” and etcd as storage backend.

4

Thank you for the information! You mention “planning to”, what are you currently using?

5

details are as follows.

Existing Vault:

aws kms for unseal keys
etcd as storage backend

New Vault(yet to create in k8s)

aws kms for unseal keys
Integrated storage as backend

6

WIth that setup there are two best possibilities, one easier than the other.

  1. You migrate (using vault operator migrate) from etcd to Raft storage.
    After this you can follow the standard backup guide to then restore the data on the new cluster.
    AWS KMS will take care of the rest of the process.
  2. Backup the etcd cluster only to then restore the etcd cluster on the new cluster.
    AWS KMS will take care of the rest of the process.

Some notes:

  • I recommend if you decide to get with the first option to backup Raft on your Raft leader. You can identify the leader in the outcome of the command vault operator raft list-peers
  • When migrating, due to that you export the data outside of Vault, I recommend rotating your encryption keys.
7

Thanks a lot for the info. I will give a try on the first option.

8

hi can you please help me on migration of below scenario.
old setup
gcp kms for unseal keys
gcp storage bucket for backend

new setup
was kms for unseal
raft for backend

I have managed the unseal part but badly stuck with backend

9

The same steps would work for you. Use the vault operator migrate command to migrate from GCP storage to Raft.

10

Thanks for your reply I have tried the same steps but getting below error, the issue may be because vault is running mode but not sure how can I stop in pod

Error migrating: error mounting ‘storage_destination’: failed to create fsm: failed to open bolt file: timeout

11

What is your mount setup? Permissions, mount type, etc. And what kind of cluster setup do you have? How are they connected?

12

here is my values.yaml file

global:
enabled: true
tlsDisable: false
injector:
enabled: true
replicas: 1
port: 8080
leaderElector:
enabled: true
metrics:
enabled: false
injector.
image:
repository: “hashicorp/vault-k8s”
tag: “0.14.2”
pullPolicy: IfNotPresent
agentImage:
repository: “hashicorp/vault”
tag: “1.9.3”
cpuLimit: “500m”
cpuRequest: “250m”
memLimit: “128Mi”
memRequest: “64Mi”
exitOnRetryFailure: true
staticSecretRenderInterval: “”
logLevel: “info”
logFormat: “standard”
revokeOnShutdown: false
affinity: |
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app.kubernetes.io/name: {{ template “vault.name” . }}-agent-injector
app.kubernetes.io/instance: “{{ .Release.Name }}”
component: webhook
topologyKey: kubernetes.io/hostname

server:
enabled: true.
secretKey: “license”
image:
repository: “hashicorp/vault”
tag: “1.9.3”
failureThreshold: 2
initialDelaySeconds: 5
successThreshold: 1
timeoutSeconds: 3
livenessProbe:
enabled: false
path: “/v1/sys/health?standbyok=true”
failureThreshold: 2
initialDelaySeconds: 60
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 3
terminationGracePeriodSeconds: 10
extraVolumes:
- type: secret
name: vault-tls
affinity: |
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app.kubernetes.io/name: {{ template “vault.name” . }}
app.kubernetes.io/instance: “{{ .Release.Name }}”
component: server
topologyKey: kubernetes.io/hostname

service:
enabled: true
dataStorage:
enabled: true
size: 50Gi
mountPath: “/vault/data”
storageClass: null
enabled: false
size: 50Gi
mountPath: “/vault/audit”
ha:
enabled: true
replicas: 3
apiAddr: null
raft:
enabled: true
setNodeId: true
config: |
ui = true

    listener "tcp" {

    address = "[::]:8200"
    cluster_address = "[::]:8201"
    tls_cert_file = "/vault/userconfig/vault-tls/vault.crt"
    tls_key_file = "/vault/userconfig/vault-tls/vault.key"
    }
    storage "raft" {
      path = "/vault/data"
    }

    service_registration "kubernetes" {}
    
    seal "awskms" {
      kms_key_id = “xxxxxx”
    disruptionBudget:
  enabled: true
  maxUnavailable: null

serviceAccount:
create: true
name: “vault-sa”
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxx:role/vault

13

it is mentioned in the documentation that vault should be offline during migration but I am running in pod so not sure how can I shutdown in pod

14

I am missing the entry for GCP storage. For Vault to be able to read out the GCP storage it first needs to read it out. Let’s start with that! :slightly_smiling_face:

15

I have migrated all the data to s3 bucket then then I logged in to my pod and created migrate.hcl file with below content
storage_source “s3” {
bucket = “xxxxx”
region = “xxxxx”
access_key = “xxxxx”
secret_key = “xxxxx”
session_token = “”

}

storage_destination “raft” {
path = “/vault/data”
node_id = “vault-0”
}
cluster_addr = “http://vault-0.vault-internal:8201”

but when I am running the migrator operator getting the below error

Error migrating: error mounting ‘storage_destination’: failed to create fsm: failed to open bolt file: timeout

16

If possible can you look into this Using Letsencrypt certificates for Vault setup on kubernetes query

17

I have migrated the data from gcp bucket to s3 bucket that’s why my source is s3 also if you can see the issue is with destination not the source, I have gone through it’s document operator migrate - Command | Vault by HashiCorp they mentioned that it’s an offline but I am doing it in k8 cluster so how can I stop vault

18

@purushothamkdr143 have you managed to migrate your data?

19

When I migrate from file storage to raft storage it creates a vault.db and a raft.db file. But to restore the data on the new cluster I need a .snap file. So I don’t understand how to migrate using option 1? Can you please explain in more detail?

20

If you are migrating file to raft outside of the final destination cluster, you would need to start up a running Vault using your temporary migrated Raft data, and make a backup snapshot from that. You could then restore that snapshot into the final target cluster.