Make vault survive redeployment

Vault
,
1

Hello vault community, can you please help me clarify one thing regarding vault with kubernetes.

I’m deploying vault on EKS using the official helm chart with KMS auto-unseal and postgres as storage backend and the process is something like this; when vault is first deployed a kubernetes job is run to initialize it, the job uses the root_token from the initialize response to authenticate itself and enable k8s auth method. This job runs in a service-account called vault-initializer.

The job reads the vault-initializer's serviceaccount token and passes it to vault to be used for token review.
Now everything works as expected and pods can authenticate with vault using their service-account tokens but now I disable the vault helm chart and reinstall it, vault comes up and auto-unseals and is already configured and initialized but when pods try to authenticate I get the error:
auth.kubernetes.auth_kubernetes_dc20e31e: login unauthorized due to: lookup failed: service account unauthorized; this could mean it has been deleted or recreated with a new token
The problem here, as I understand it, is that my vault-initializer serviceaccount has been recreated with a new token and vault can’t authenticate other pods using the old token.

So my question is really this, how do I now update the vault configuration to use my new service-account token for token_review? I cannot authenticate with vault using serviceaccount anymore and the job that had the root token to authenticate is dead and gone. Am I missing some steps here that would simplify this or am I doing something fundamentally wrong? Should I be saving my root token somewhere for this purpose, am I forever locked out if I lose it? What is the purpose of the auto-unseal if the rest of the vault configuration cannot survive redeployment?

2

In general you shouldn’t use the root token on an ongoing basis. Good
practice would be to use the token for as short a period as possible
(e.g. to enable the auth method) and then revoke the token. If at a
later point more changes are needed you can create a new root token from
the recovery keys (revoking it again once used).

So in this case if you are wanting to update the k8s auth method
configuration you need to login with something that has write permission
to the correct path. A root token could be used if you don’t have
anything else - create the token by using the recovery keys.

However I’m not understanding why you disabled the chart? Once Vault is
installed you should just need to update it (e.g. for new versions or
configuration) but you shouldn’t totally remove it. In general I
wouldn’t expect a chart that is totally removed and then reinstalled to
work with anything created by the previous install, as particularly for
security systems the chart may have created certificates, keys, etc.
that would be stored within resources such as secrets. Totally removing
the chart would remove those resources, losing the ability to access any
data previously produced.

In the case of desaster recovery you would cover this via standard
backups. Upon initialisation of Vault it is important that you securely
store the recovery keys to allow you to create root tokens in the
future. I would generally expect the initialisation process to be done
manually, rather than being automated.

1 Like