Hi, I’ve setup AWS KMS auto-unseal according to the docs with vault running on a couple of ec2 servers in auto-scale groups. Everything works.
For my setup though, due to our internal cyber policies, we cannot run apps/processes under the default “ec2-user” account which can sudo to root. So we’ve created a local account via “useradd”. Believe this is recommended as well.
In any case, all functions work… accept the final stage of the root recovery process. I start a new root recovery process. i can key in the various keys until the final key where Vault tries to then generate a new root token - we get this error:
root generation aborted: unable to authenticate: recovery key verification failed: failed to decrypt encrypted stored keys: error decrypting data encryption key: ExpiredTokenException: The security token included in the request is expired
The way we’ve setup our vault systemd startup script is to curl the AWS meta-data/iam/security-credentials/role endpoint to obtain the aws access key, secret key and sessoin token for the local “vault” user. we then export this before starting up vault.
This is all in the systemd startup script. So I was guessing the session token expired, so when I restart all my vault process, Voila it works!
So my question here is how do we design this to enable recovery without needing to restart vault? I was under the impression if one uses the AWS SDK properly, it will rotate the session token, but it doesn’t seem to be happening!
