Hi Folks,
My Current Setup:
I have deployed Vault in AWS EKS using helm 3. The configuration is made HA using Raft config with 3 nodes backed by EBS persistence volumes. The Vault is also configured to use AWS KMS for auto unsealing on restart.
My Questions:
Question I have is even with this HA setup, do we need to have a backup of secrets in text files on local or a secure location with password protection. Or can we completely rely on the HA setup?
What happens when a Multi AZ, Raft HA Vault setup gets destroyed due to a natural or human disaster?
Current Solution
Current plan to achieve recovery on a complete destruction of the cluster is to have a copy of all static secrets in files on local or remote repository and have shell or ansible script which can setup the current set of secrets, policies and roles to a new Vault cluster within minutes and make it available for the applications to use.
File in local or repository can be encrypted or password protected using tools like Ansible Vault.
One other idea which I have not tried is using the Raft Snapshot, can we restore the raft snapshot from old Vault cluster into a completely new vault cluster? (Have not tried this yet)
Need Advice:
Please let me know, if you have ever had this question on your mind or if you already considered this as a possibility how did you tackle the problem?