Vault Raft Recovery after power outage
I am having some issue with my current setup with Vault and kuberentes.
Setup :
Microk8s HA mode
Vault HA mode
Rook-ceph : HA mode
metallb : setup
Recently we had a power outage, and the vault leader went into weird state and after restart of the pc, the raft db size is 16mb and stuck at initialization as false. Other instances are initialized true and total shares 0 but the other 2 instances shows initialization true and total shares as 5 but all instances shows removed from cluster :false and vault version shows as 1.20.4. I am trying to recover the raft data and bring back to the previous state.
Here are the options i have tried until now
1. Isolating the first instance and bypass the https, tried to login and other operations like unseal, on 0 instance throws 503 errors
3. Unsealing on other instances throws 400 error
4. checking the pvc's and pv's for any lost data, nothing found
5. Vault Proxying on second instance through UI showed unseal keys but when i entered 3 it "failed with wrong cipher authentication"
6. Proxying port forward on first (vault 0) instance through UI , asks for rejoinging the cluster when i entered all the keys, certs and every information it simply throws error "failed to join the cluster".
Just FYI: I think i have lost the previous node-ids and vault has re-initiated new ids with instance names.
Can someone please help me with this on how to recover the raft data and make it work again.
Also need suggestions on how to avoid these kind of situations in future.
1. some suggested to cron jobs.
2. some suggested tools to auto backup raft data.
but i am not aware of those methods. Can someone please suggest if tehre are any other ideas or if those 2 only which is the better one in production and instructions to implement it.
Thanks for going through this question. If you have any questions please let me know and i will try to reply back asap. I spent 2 days with no luck.
On vault 0 instance when i try to run in recovery mode it throws "failed to create fsm: failed to open bolt file: timeout" error.