I’ve some questions on backup and restore Vault cluster backend, the storage backend is a consul cluster.
My plan is to save a vault data backup daily with a cron, then if someone or application delete some vault entries, we will use the daily backups to restore.
Since the data storage backend is consul cluster, I’m using ‘consul snapshot save <daily_snapshot_file>’ command to do the real work.
I’ve read the doc at Vault Data Backup Standard Procedure | Vault - HashiCorp Learn, Though, confused on the preparation list, especially this sentence ‘Bring your Vault and Consul clusters back online following the circumstances that required you to restore from backup’
I don’t have much experience on vault/consul. So to play it safe, before running the above data backup command with consul, I seal the every member of the vault cluster.
This works fine, but during the time period when vault cluster is sealed, vault clients fails to get/put data to vault. Some time when there are a few tens of GB data in the vault, the backup time can be pretty long, and so vault clients are adversely affected.
Is it safe to run ‘consul snapshot save …’ command when the vault cluster is unsealed? If not, is there any alternatives, say toggle some options/switches to partially locking vault secret paths, similar to MySQL DB/Table/Rows locking?
Currently, in fear of vault data corruption, My backup procedure is:
1, seal all server members of vault cluster
2, on the first vault server member, run ‘consul snapshot save <snapshot_file>’
3, unseal all server members of vault cluster
4, encrypt the snapshot and save it to offsite.
Restore doesn’t happen often, but the steps are very similar to the above.
Please advice, Thanks,