I’m trying to simulate possibility of migrating vault instance using the AWS KMS (customer provided key material) to a different KMS key/aws account.
So far - I have created a KMS Key “aaaa-…”, set up vault and configured auto unseal. It works great so far.
Now I create a new KMS Key “bbbb-…” with the same key material and disabling “aaaa-…” (to simulate total unavailability of the key with the original id).
Now if I change the config to point to “bbbb-…” and start vault it still logs:
failed to unseal core: error="fetching stored unseal keys failed: failed to encrypt keys for storage: error decrypting data encryption key: DisabledException: arn:aws:kms:ap-southeast-2:xxx:key/aaaa-... is disabled."
Even though config only points to the new “bbbb-…” key vault somehow know about “aaaa-…”
The backend storage used is S3.
Is it technically possible to do what I’m trying to do?
I can see references to the original key id at core/recovery-key and core/hsm/barrier-unseal-keys and from reading code it looks like it’s impossible to avoid decrypting it using the original key.
Okay, after debugging vault I have realised that it’s how aws kms symmetric encryption works: it puts the original key id into the encrypted data payload automatically.
Which means - there is really no way to migrate to a new key id even if you know the key.
What’s the DR scenario for running aws kms unseal then?
There are three levels of keys with Vault: the key to protect the storage, the key to protect the storage key(s) (i.e., the master key), and the seal key. The seal key is a completely separate layer of cryptography that needs real-world processes and procedures around it to ensure your Vault deployment is properly protected.
Disaster Recovery (DR) is a feature of Vault Enterprise, and, in a typical scenario, it would be used to stand up a secondary cluster that shares all the key material of your original (called primary) cluster. Without DR, the only way to ensure you can’t lose your seal key – and therefore, your Vault – is to keep a backup of it. Unencrypted? Encrypted with another key? But then, how do you protect that key? As you’ve discovered, none of those non-Enterprise scenarios are very appealing.
Oh, well, in your situation, it would be the customer that would have to backup this key material they are providing; you’re absolutely right: you have no ability to do so, in this case, as I understand it.
Okay, I probably was not clear enough: even if you have key material - it would not help you: you cannot re-create the same CMK out of it, it would have different key id. So AWS KMS Decrypt would fail without even trying to decrypt.
So, in short: if aws kms cmk is lost (eg: it was removed), then it would be impossible to unseal anymore. Or restore from a raw storage backup.
I wouldn’t be able to sleep well if I knew I run a core security system that is impossible to backup/restore
This isn’t really anything to do with Vault per-se. This is an AWS situation, which would presumably be the case for anything using KMS (I’ve no idea what the general recommendation for DR is from AWS).
As mentioned Vault does offer the DR option using Enterprise which allows for running DR instances in other regions/clouds/locations which works regardless of the unsealing method you use.
if you use a different Vault cluster with transit engine for auto unseal, backing up that second Vault cluster is definitely possible. I am not familiar with the other options, maybe someone else can tell you more.
DR with Vault Enterprise is not directly related to backing up auto unseal keys. DR is for a secondary cluster that can take over immediately if the primary fails.
The auto unseal keys are in a system unrelated to Vault (AWS KMS in this case). Therefore it depends on that third party system if they allow you to export the keys. I can’t tell you if it makes sense you can’t switch to a different KMS key with the same key material because I lack experience with KMS.
How? Imagine you lost everything (but backups). What exactly you initially should have backed up to now spin new vault instance(s) from scratch from backups?
Let’s say you have Vault cluster A that is used to auto unseal Vault cluster B. Assuming you have backups for both clusters and the unseal keys for cluster A, you could restore everything.
Restore cluster A from backup and unseal with unseal keys
Recreate cluster B from backup. It can use cluster A to auto unseal.
@jlj7 I am not sure I follow. If the auto unseal key is lost, neither the primary nor the secondary cluster can be unsealed once they are sealed. Maybe it’s possible to use different auto unseal keys for the primary and secondary cluster? I am not sure, in that case it would be helpful.
@zerkms The scenario I described uses a second Vault cluster for auto unseal. That second cluster has to be unsealed somehow, too. I assumed unseal keys for the second cluster.
Okay, it looks like for vault to be safely backup’able when using just aws kms (and most likely majority of other auto unseal mechanisms) - it should be redesigned to support 2 unseal mechanisms simultaneously for one vault instance: say first auto unseal aws kms, the second - shamir keys stored offline for DR purposes only.
I’m sure it would never be implemented, and until then - using aws kms is a recipe to lock you down in case of an opertational mistake (or malicious actions).
