AWS KMS Expired Token error on generating new root from recovery key

Hi, I’ve setup AWS KMS auto-unseal according to the docs with vault running on a couple of ec2 servers in auto-scale groups. Everything works.

For my setup though, due to our internal cyber policies, we cannot run apps/processes under the default “ec2-user” account which can sudo to root. So we’ve created a local account via “useradd”. Believe this is recommended as well.

In any case, all functions work… accept the final stage of the root recovery process. I start a new root recovery process. i can key in the various keys until the final key where Vault tries to then generate a new root token - we get this error:

root generation aborted: unable to authenticate: recovery key verification failed: failed to decrypt encrypted stored keys: error decrypting data encryption key: ExpiredTokenException: The security token included in the request is expired

The way we’ve setup our vault systemd startup script is to curl the AWS meta-data/iam/security-credentials/role endpoint to obtain the aws access key, secret key and sessoin token for the local “vault” user. we then export this before starting up vault.

This is all in the systemd startup script. So I was guessing the session token expired, so when I restart all my vault process, Voila it works!

So my question here is how do we design this to enable recovery without needing to restart vault? I was under the impression if one uses the AWS SDK properly, it will rotate the session token, but it doesn’t seem to be happening!

Hi @stevenpeh – you should remove the curl’ing of the credentials from the EC2 instance metadata endpoint and not set any credentials in environment variables. When you set credentials in environment variables, the AWS SDK will just always use those and not go to the instance metadata service for refreshed credentials. If there are no credentials explicitly provided, then the AWS SDK will reach out to the EC2 instance metadata service on its own and rotate the credentials when needed.

Wow thanks for the quick response! Just tried that and it seem to be unsealing and the root re-gen works!

Should add this narrative into the docs as pretty much one of the initial steps is to set those variables!

Awesome, glad it worked out for you!

I took a look at the docs and agree they’re kind of confusing. The docs are actually all on GitHub, and I’m sure the Vault team would welcome a contribution to improve them if you wanted :slight_smile:

1 Like