Vault Storage Migration in running state

Hello HashiCorp Community

I want to migrate from an etcd backend to integrated storage. The documentation states the following: “This is intended to be an offline operation to ensure data consistency […]”. I understand that it is “intended” but not “mandatory”. Is that correct?

My problem with the offline mode is this: I tested the migration by spinning up a second Vault cluster from the etcd backup of the productive cluster. Then I stopped the cluster as described in the doc and proceeded with the migration which took appr. 4 hours. Although I could work with a “Read Only”-Vault for 4 hours I cannot work with an offline cluster for 4 hours.

That’s why I’m wondering what exactly the reason of the offline procedure is. May I loose data that is being written during the migration or could something more serious happen?

Thanks for your answers

Luana

It is mandatory to migrate offline.

This is because the migration is just a sequential copy of all keys in the data store, and if keys are being added, deleted and modified whilst the copy is ongoing, the result will be an inconsistent mixture of data of different ages, which will result in undefined malfunctioning behaviour.

1 Like

Sadly there are no on-line migration options at the moment but with a storage that takes four hours to migrate Integrated Storage is a solid choice.

The only option I can think of is to write a script to backup the policies, make all the policies read only (“read” and “list” only). Duplicate the etcd instance (backup & restore?) … Then use the new etcd as your migration source to the new Vault+IS instance. Replace the Vault instance, restore the original policies. You’d lose any new tokens/leases generated but that’s the least amount of down time I can think of.