Safe to have AWS KMS rotate outside of vault?

Vault
1

Hello,

Is it ok if AWS KMS is setup to rotate the CMK automatically that is used in auto-unseal outside of running any sort of vault commands, such as by someone in AWS UI manually or configuring it in AWS to rotate automatically every year, or would these scenarios cause issues and brick everything on the vault side?

2

Hi!

We haven’t explicitly tested this scenario. It depends on whether items that were encrypted using the key before rotation could still be decrypted using the key after the rotation. I did a quick scan of the AWS docs on it and I see that the key ID remains the same and the backing key changes. It doesn’t explicitly state whether items encrypted with the previous key would still succeed in decryption. The new cryptographic key would obviously be unable to decrypt previous material, but AWS may save old versions of the key and try them, I’m not certain.

This is a long way of saying - :slight_smile: you’d have to test to find out.

A quick way of testing would be by running Vault using something like $ vault server -config=/path/to/config.hcl -dev -dev-root-token-id=root, where the content of your config would be something like:

storage "file" {
  path = "/tmp/vault/data"
}
disable_mlock = true
ui = true
listener "tcp" {
  address     = "127.0.0.1:8200"
  tls_disable = 1
}
api_addr = "http://localhost:8200"

seal {
  "awskms" {
    region     = "us-east-1"
    access_key = "foo"
    secret_key = "foo"
    kms_key_id = "foo"
  }
}

Then you’d want to rotate the KMS key, stop Vault, and restart it to see if it comes up again.

-Becca

3

By the way, if you do decide to test it, please do post the results because it might help other members of the community.

Thanks again!

4

Thanks, I’m sure we’ll be testing it soon enough, even though, I’m not crazy about potentially bricking vault atm and having to redo the cluster we have.

The vault docs at the bottom here do seem to state that manual and automated KMS rotations are supported: https://www.vaultproject.io/docs/configuration/seal/awskms.html

I’ll reply whenever we do test if we’re hosed or not. Crossing my fingers that worst case we’d maybe have to update vault server configs and restart some nodes for the newly rotated key to be picked up and work.

5

wisdom_of_the_ancients

Any updates on this one ?

6

The basic principle here is:

It’s fine for your cloud KMS to create new key versions, and start using them for new encryptions.

It’s very much not fine for it to take offline or expire old key versions, as there is no guarantee that Vault has stopped needing them to decrypt old data.