Migrate vault from Non-HA to HA (Raft based backend)

Vault
, ,
1

Can I migrate from
Non-HA Vault (File based Storage)
to
HA Vault (Raft based Storage) ?

If its possible, can you please share the doc for this.

2 
$ cat vaultmigrate.hcl
storage_source "file" {
  path = "/var/lib/vault-file/"
}
storage_destination "raft" {
  path = "/var/lib/vault/"
  node_id = "node01"
}
cluster_addr="https://node01:8201"

$ vault operator migrate -config=vaultmigrate.hcl

Source: Migration from file to raft fails for large indexes · Issue #12487 · hashicorp/vault (github.com)

Official documentation: operator migrate - Command | Vault by HashiCorp (vaultproject.io)

3

@Wolfsrudel has got the 2nd step, but first you need to establish a RAFT cluster. Have at least 3 nodes in your cluster so that raft is up and running, then you can move the data from file storage to raft storage.

4

I don’t think this is correct.

According to operator migrate - Command | Vault | HashiCorp Developer (emphasis mine):

The operator migrate command copies data between storage backends to facilitate migrating Vault between configurations. It operates directly at the storage level, with no decryption involved. Keys in the destination storage backend will be overwritten, and the destination should not be initialized prior to the migrate operation. The source data is not modified, with the exception of a small lock key added during migration.

This is intended to be an offline operation to ensure data consistency, and Vault will not allow starting the server if a migration is in progress.

I think the segments I’ve bolded say that you should NOT be establishing a running cluster before migration.

5

@maxb there are two different backends, setting up a raft cluster has nothing to do with the data store.

6

My Non-Ha Vault (File based storage) exists as StatefulSet in Kubernetes (deployed via Hashicorp Helm).

This is what I did.

  1. Updated my ConfigMap from File to Raft
apiVersion: v1
data:
  extraconfig-from-values.hcl: |-
    disable_mlock = true
    ui = true

    listener "tcp" {
      tls_disable = 1
      address = "[::]:8200"
      cluster_address = "[::]:8201"
    }
    storage "raft" {
      path = "/home/vault/"
[...]
  1. Deleted the Pod to pickup the config and increased the replicas in STS to 3
  2. Placed the file - /home/vault/vaultmigrate.hcl in Pod
storage_source "file" {
  path = "/vault/data"
}
storage_destination "raft" {
  path = "/home/vault/"
  node_id = "vault-0"
}
cluster_addr="http://vault-0:8201"

However, Migrate does not work.

kubectl exec -ti vault-0 -- vault operator migrate -config=/home/vault/vaultmigrate.hcl
Error migrating: error mounting 'storage_destination': failed to create fsm: failed to open bolt file: timeout

Can this work in Pods?
Please correct me if I the above steps are not in order.

7

What are these two different backends you speak of?

Are you referencing the possibility of using Raft only for ha_storage whilst a different storage backend is configured for storage? I do not think that’s what the original poster is asking about.

No, the purpose of the Raft cluster is to be a replicated data store.

8

I have not tried it in pods, but I can’t imagine they would miss that scenario. Is your new raft cluster up and running and can it be reached from where you’re executing your migrate command? Remember that the cluster has been available and healthy for you to migrate data to it.

9

I can see some problems in your procedure:

You say you’re using the Hashicorp Helm chart, but you also say you’re manually editing a ConfigMap - you don’t mention anything about updating the values.yaml passed to Helm?

You seem to be trying to run the migration after you’ve already started Vault using the new configuration - but storage migration is something you do whilst Vault is not running.

It’s not a good idea to be moving the Vault storage into the local filesystem of the pod (which I assume is what /home/vault is?), as this would mean if all your pods ever exit, all your Vault data has been lost. Ensure you use a path that is a PVC.

It might be best for you to first practice deploying a new Vault cluster using Raft storage, using the Helm chart. Once you’ve got that proven, you can start thinking about how you’ll migrate data. I can’t see any official docs for how to do this in a Kubernetes/Helm deployment, so be aware you’ll probably need to improvise a fair bit!

I imagine it might look something like this (this is me just designing a new procedure on the fly… please test it before you rely on it!):

  • Delete your StatefulSet
  • Run a custom Pod definition that mounts your PVC, but doesn’t start a Vault server. In this, you can run your vault operator migrate command
  • Delete that temporary pod, and redeploy a new StatefulSet using Helm.
10

This is incorrect advice.

I quote from operator migrate - Command | Vault | HashiCorp Developer :

Vault will need to be offline during the migration process. First, stop Vault.