How to unseal cluster B (how to perform the recovery), when cluster A is down and cannot be started?
To be honest, I’m a bit disappointed with >Hashicorp<
The product seems to lack proper documentations - there are some examples, but you have to “discover the wheel from scratch” in many cases.
Maybe if you know the product, things are obvious, but it is definitely not intuitive.
There are no ‘best practices’ - 3-4 random articles, that you have to glue together if you’d like to run a sane environment.
EG - looking for a documentation of
2 HA (3-5 nodes) clusters
where at least one has autounseal (preferred mutual autounseal with shamir recovery as a manual “planB”)
a scenario for cluster-with-autounseal transit recovery (topic of this post)
TLS example (and guidance with example for certificates/credentials replacement - rolling update, nodes replacement or full cluster stop?)
kubernetes integration with vault - best practices on certificate rotation
best practices on vault ACL models
best practices on backup/restore of your transit key
any description of 2 mutual-autounseal clusters (best practices, deadlock recovery, any option for a recovery with shamir? external shamir cluster?)
How do you run your clusters, if you cannot answer to that…
I feel you might be setting up for failure or frustration with your intention to use 2 clusters to mutually unseal them. This isn’t an intended use case…
Using transit to unseal another cluster expects the transit cluster is highly-available. If you lose this cluster, you won’t be able to unseal. If you are only using it for unseal, it can go down and will not seal the other cluster or do anything bad immediately. But, when your cluster leveraging its transit endpoint needs to be unsealed again, it will not work. If you lose your unsealing-transit-cluster, you’ll have to recover the cluster that depends on it, if you can’t get it back online.