Hey, over the holidays one of my vault backups stopped verifying, and I’m trying to work out why. This is the error I’m getting:
fetching stored unseal keys failed: failed to decrypt keys from storage: error decrypting data encryption key: AccessDeniedException: The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access.
I’ve verified that
A. Keys exist
B. The AK/SK that Vault is using has full access to those keys
C. The AK/SK can run the encrypt/decrypt/describe commands via the AWS cli
I have no clue what’s up. Other vault backups restore correctly, this one (and only this one) is failing. Is there a way to increase the log levels past trace, or add some AWS-specific logging?
First I spin up a vault server, init, unseal, and then restore the backup to it. Then I reboot the server, and it fails to unseal. I’ve ran ‘aws kms encrypt’ and ‘aws kms decrypt’ commands before and after the vault server command to verify that the is accessible by that user, and those commands run successfully.
Any help on how to debug this kind of issue? Everything is configured via env vars, per best practices.