Vault snapshot restore process

Hi All,

I am in a new environment where i have to migrate Vault data from one environment to another. I took a snapshot from source and restored in the destination. Vault went to sealed state and I have to unseal it using the source recovery keys.

In the source the keys were stored in SSM as encrypted format. I was wondering how to decrypt it because the public keys were used during vault process and i am not sure how -pgp-keys option works.


GPG_KEY_PATHS="/home/app/keys/sealer1,/home/app/keys/sealer2,/home/app/keys/sealer3,/home/app/keys/sealer4,/home/app/keys/sealer5"

NONCE=$(vault operator rekey -target=recovery -init -pgp-keys="${GPG_KEY_PATHS}" -backup -key-shares=5 -key-threshold=3 | grep 'Nonce' | awk '{print $2}')

After this step the encrypted keys were stored in parameter store.

Now I have those 5 public certificates and i can import to gpg but i am not able to decrypt them because the gpg doesn’t have the secret keys.

Kindly advise how the options -pgp-keys and-backup works.

Like where the backup will be stored or anyway I can decrypt the recovery keys.

Thanks you so much in advance.

Just a note : - When i run the above in local docker-compose test env I am able to decrypt those keys and I noticed the gpg lists secrets and so i understand gpg stores the private keys locally.

WARNING: You cannot unseal Vault using recovery keys alone. (Personally, I think this makes them dangerously misnamed.) You still need access to the original auto-unseal method.


The only way to decrypt these recovery keys is to have someone who possesses the PGP secret keys do it.

1 Like

Well yeah … that’s the whole purpose of doing it via encryption. If the same person who rotated could unencrypt the keys what would be the purpose of the encryption in the first place.

So that’s why Vault gives you two choices:

  • Non-encrypted … where the operator takes the resulting new keys and distributes them.
  • Encrypted … where each key holder provides their public key … and get an encryption of their portion of the master key encrypted using that key. They would have to unencrypt the key by using their private key.

Probably because you gave Vault your public key only and you have your private key. Therefore they’re all encrypted with your key and you can read those.

1 Like

Hi @aram,

Thank you so much for the response. Can you tell me how the option -pgp-keys=“${GPG_KEY_PATHS}” -backup is used here. I understand that i have the public keys but what happened to the private key is not clear. I do not see anyway the private keys exported and saved somewhere. By any chance will the -backup option stores the private keys in vault and if yes how to get it. Kindly advise.

Thanks.

No, the -backup option does not store the PGP private keys in Vault. Vault never sees the PGP private keys at all.

The -backup option stores a copy of the new Vault unseal/recovery keys (whichever are being created by the operation) which are encrypted with the PGP keys, to Vault storage, so that each PGP key owner can their encrypted Vault key from the Vault API.

Hi @baranikannan – I think there is a misunderstanding of how public/private key encryption works. This is the same process that SSH keys are based on.

You generate a pair of keys a private one and a public one. You can share the public one with anyone on the internet. When someone wants to send you a secure message, they can use your public key to encrypt that message. When you receive the message your private key (which never leaves your system) can be used to decrypt the original message. Vault is using this to encrypt the keys, so the only people who have the private key is the original person who sent you their public key.

Now as far as the backup option this is just a delivery safety. When the key is encrypted instead of you emailing or SMS or DM-ing that key to the receiver Vault also stores that key in Vault itself.

The proper best-practice calls for the operator to immediately send out the keys and delete any trace of them. Well what happens if one of the keys is lost in transit or the receiver deletes the message mistakenly? What happens if the operator machine crashes or the network goes down while you’re generating the keys? Well now there is backup of that encrypted key in Vault itself where it can retrieved. That’s the backup option… it’s “oh no” moment right after key generation/delivery.

Hi @aram,

Thank you so much for the response. I finally identified the problem in other way. Actually the Vault cluster is using awskms so i thought of giving permission to the source environment’s kms and it worked. I do not have much understanding on how these keys works and now i am able to bring up the cluster. But I have to find a way to keep those keys safe by doing rekey(but without existing key not sure how to do that using awskms) for future use. Thanks again.

Regards
Barani

When using autounseal, your “shamir” keys turn into “recovery” keys. The actual master key is now stored in AWS KMS. The recovery keys are still needed to generate a new root token, or any DR operations.