We are a small team that wants to set up a HA cluster for Vault and we have access to 1Password’s Vault. What is the best practice for where to securely store Vault key shares and initial root tokens for each of the nodes in the cluster? We want to understand how to unseal cluster nodes when they come back up after a failure? Is there a way to make it so that the unsealing of a node when it comes back up in an HA cluster can be done automatically without manual intervention or third party cloud providers (AWS, Azure, etc). We’re hosting our servers at OVH.
A lot depends on your organisation’s specific security requirements. i.e. What threats are you aiming to protect against?
e.g.:
- An external threat gaining network access to your servers?
- An insider threat, like an employee with administrative privileges going rogue?
- Someone stealing a physical disk or exfiltrating a backup disk image and attacking it offline?
Once you have a threat model defined, you can use it to evaluate whether options are or are not OK.
e.g. If you store your unseal key locally on the disk of the Vault server, it’s easy to inject it into Vault in a startup script, but anyone with access to the disk can decrypt your Vault.
If you protect your key in some kind of key management system accessed over the network, then you’ve protected against someone stealing your server’s disk and attacking it offline, but anything which can authenticate to the key management system can still decrypt your Vault.
The highest level of protection comes from a locked down system, such that no-one is allowed to log in at all without a second (or even more!) people watching their every action, and all of these people collectively performing agreed-upon-in-advance integrity checks, before supplying their own key shares.
There are any number of ways you can go about this. Depending on how critical your vault infrastructure is your team/company.
I’m going to assume this is a corporate environment with some dependencies on your Vault. A lot of what’s below can be tuned down or ignored if not.
Ideally you want to revoke your root key after your initial setup. You can always generate a new one if needed by using your keys but you shouldn’t leave a root key laying around anywhere – even securely.
Let’s start by saying you don’t unseal “nodes” you unseal the cluster once. Once Vault has it’s encryption key assembled it’ll communicate that to the nodes via internal communication. In Kubernetes there are cases when each node needs to be unsealed but it doesn’t sound like you’re using kubs.
You can and “probably” should setup an auto-unseal. Last thing you want is for Vault to restart and someone needs to be woken up to start the unseal process because the provider decided it was a good time to do maintenance. The process of setting up auto-unseal changes your “unseal keys” to “recovery keys”. You still need the recovery keys for various activities, but they are no longer used to “unseal” Vault – if you ever lose your cloud unseal keys (wherever they maybe stored) you’re going to be in real trouble.
That said – In best-practice – it’s a called an unseal ceremony (although as stated they’re not really unseal keys if your using auto-unseal) that should be well documented and practiced (we do it twice a year – plus, as key holders change).
A couple of good practices that we have come up with: We generate a lot more keys than we need – really as many as we have people. So 11 keys are generated but we only need a quorum of 3 keys to complete. Those 11 are divided up into East and West (world) and a pagerduty or opsgenie DL is created for those people who can be pinged for their key (which they can store in whatever secure method they can).
Reference: