Hello vault community, can you please help me clarify one thing regarding vault with kubernetes.
I’m deploying vault on EKS using the official helm chart with KMS auto-unseal and postgres as storage backend and the process is something like this; when vault is first deployed a kubernetes job is run to initialize it, the job uses the
root_token from the initialize response to authenticate itself and enable k8s auth method. This job runs in a service-account called
The job reads the
vault-initializer's serviceaccount token and passes it to vault to be used for token review.
Now everything works as expected and pods can authenticate with vault using their service-account tokens but now I disable the vault helm chart and reinstall it, vault comes up and auto-unseals and is already configured and initialized but when pods try to authenticate I get the error:
auth.kubernetes.auth_kubernetes_dc20e31e: login unauthorized due to: lookup failed: service account unauthorized; this could mean it has been deleted or recreated with a new token
The problem here, as I understand it, is that my
vault-initializer serviceaccount has been recreated with a new token and vault can’t authenticate other pods using the old token.
So my question is really this, how do I now update the vault configuration to use my new service-account token for token_review? I cannot authenticate with vault using serviceaccount anymore and the job that had the root token to authenticate is dead and gone. Am I missing some steps here that would simplify this or am I doing something fundamentally wrong? Should I be saving my root token somewhere for this purpose, am I forever locked out if I lose it? What is the purpose of the auto-unseal if the rest of the vault configuration cannot survive redeployment?