Performing restore from snapshot invalidates existing auto-unseal recovery keys

Vault
1

I am using self-managed Vault 1.12.2 w/ integrated storage and AWS KMS auto-unseal.

I notice that any existing root token and/or recovery keys are invalidated upon doing a restore from snapshot in a test exercise, following Vault restore tototial.

Two questions:

  • Is this intentional behavior?

  • Is there is any way to generate new recovery keys after doing a restore? In general, I revoke root tokens and operate day-to-day via a few auth backends such as OIDC. But I have always retained recovery keys for an emergency and it is bad to think I would not have recovery keys for any future break-glass scenarios following a restore

Below are steps to reproduce:

Case 1) Root login

  • In a test exercise, I deliberately got my Vault cluster in a bad enough state that I needed to vault operator init (using KMS auto-unseal).
  • I did not create any additional auth backends
  • I then logged in with the resulting root token
  • I then ran vault operator raft snapshot restore -force path/to/snapsnot.snap on cluster leader
  • When I tried to log in again with root token to view a secret, I got a 403, eg
> vault login
Token (will be hidden):  hvs.redacted-root-token-that-worked-pre-restore
Error authenticating: error looking up token: Error making API request.

URL: GET https://my-vault-addr.dev/v1/auth/token/lookup-self
Code: 403. Errors:

* permission denied

Outcome: I was stuck unable to auth to my cluster and therefore unable to access my restored data

Case 2) SSO login

I then got the cluster back into a sealed, unitialized state, and tried:

  • Run vault operator init. Safeguard root token + 5 recovery tokens
  • Create additional auth backends and log in with SSO auth backend
  • Run vault token lookup on root token - token is valid
  • Run vault operator raft snapshot restore -force path/to/snapsnot.snap
  • SSO login again
  • Run vault token lookup on root token - token is now in invalid, eg
> vault token lookup hvs.redacted-root-token-that-worked-pre-restore

Error looking up token: Error making API request.

URL: POST https://my-vault-addr.dev/v1/auth/token/lookup
Code: 403. Errors:

* bad token

Outcome: Able to view secret via SSO login, but root token is invalid if I needed it for any more elavated actions

Case 3) Rekey and/or generate-root

I then got the cluster back into a sealed, unitialized state, and tried:

  • Run vault operator init. Safeguard root token + 5 recovery tokens
  • Go through a rekey via vault operator rekey -init -target=recovery. This works. This is so I can establish a base case of working rekey steps
  • Go through a vault operator generate-root -init. This works. This is again to establish a base case of working generate-root
  • Revoke original root token generated via vault operator init
  • Log in with SSO auth backend
  • Run vault token lookup on root token generated via vault operator generate-root - token is valid
  • Run vault operator raft snapshot restore -force path/to/snapsnot.snap
  • SSO login again
  • Run vault token lookup on root token - token is now in invalid
  • Run vault operator rekey -init -target=recovery. Rekey fails, eg
Error posting unseal key: Error making API request.

URL: PUT https://my-vault-addr.dev/v1/sys/rekey-recovery-key/update
Code: 400. Errors:

* recovery key verification failed: recovery key does not match submitted values
  • Run vault operator generate-root -init. This also fails, eg
Error posting unseal key: Error making API request.

URL: PUT https://my-vault-addr.dev/v1/sys/generate-root/update
Code: 400. Errors:

* root generation aborted: unable to authenticate: recovery key verification failed: recovery key does not match submitted values

Outcome: Able to view secret via SSO login, but unable to rekey or generate new root token

2

This is all functioning as expected … restoring a snapshot restores the full state of the cluster from when the snapshot was created including tokens (root and otherwise) and recovery key active in the source cluster at that time.

Additionally, the snapshot requires access to the original AWS KMS auto-unseal method that was used in the source cluster when the snapshot was created.

You may already know this, but just to make sure: a snapshot cannot be unsealed with recovery keys alone - the AMS KMS unseal key must still be available, or the data is lost.

1 Like
3

Thank you! That makes perfect sense. Really helpful. So really the answer is to just preserve the original recovery keys I had prior to breaking Vault (when snapshot was created) and to throw away the new keys I generated just before restore.