After upgrading to Kubernetes 1.21, kubernetes authentication request to vault fails with "permission denied"

We recently upgrade to the K8s version to 1.21. After upgrading, clients are unable to authenticate themselves with kubernetes authentication.
we have configured kubernetes authentication as suggested in the document: https://www.vaultproject.io/api/auth/kubernetes
The issuer field is also set. We have tried both options: setting issuer field and disabling issuer validation.
Immediately after configuring vault, the authentication works fine.
It starts failing with the error “permission denied” after vault pod is restarted.
Vault logs show the following reason for rejecting the request:

login unauthorized due to: lookup failed: service account unauthorized; this could mean it has been deleted or recreated with a new token

It starts to work again, if configuration commands are executed again.

1 Like

I think I am experiencing the same issue, see also https://github.com/hashicorp/vault/issues/11953#issuecomment-916828227 for a reproducible example.

Everything works fine up until a restart of the vault pod.

Thanks for raising this. Judging from the linked GitHub issue, I think what’s probably happening here is the k8s token that k8s auth is configured with gets deleted when the vault-0 pod is deleted because projected volume tokens are tied to the lifetime of their associated pod.

K8S_TOKEN=$(kubectl exec vault-0 -- cat /var/run/secrets/kubernetes.io/serviceaccount/token)
...
vault write auth/kubernetes/config token_reviewer_jwt="$K8S_TOKEN" kubernetes_host="$K8S_ADDRESS" kubernetes_ca_cert="$K8S_CA" issuer="$K8S_ISSUER"

# This line then deletes the token saved in K8S_TOKEN, meaning Vault no longer has valid credentials to query the Kubernetes API with
kubectl delete pod vault-0

Broadly, you could solve this in one of two ways:

  1. Use a long-lived token, i.e. use the service account’s default token which is stored in a k8s secret instead of the ephemeral token that gets mounted in the pod
  2. Don’t set token_reviewer_jwt, and instead apply the system:auth-delegator role to the service accounts logging in to Vault. Without token_reviewer_jwt set, k8s auth will use the JWT passed to it during login.

Unfortunately, we do have a lot of documentation/tutorials that assumes the token in Vault’s own pod is long-lived. I’ll raise this internally and we’ll review how we want to address this for our own documentation.

2 Likes

Thanks for the reply! The first workaround I’d like to avoid (as it kinda feels wrong, as in “Yeah login happens with this short lived token, but btw. vault itself uses a static one to check for validity”)

The second one works if I also set disable_local_ca_jwt=true - I do not know if this is intentional? I’d assume having that on default (false) would already do exactly that - check for validity with the local token if token_reviewer_jwt is unset (and therefore also not needing an additional rolebinding for all serviceaccounts accessing vault)

It would be nice if the issue is fixed in vault. An option could be provided so that it can be configured to always use the current local token available in the pod instead of storing the JWT token available/provided when kubernetes authentication is configured.
With the current limitation, however, I prefer the first approach. Second approach requires rolebinding for all service-accounts accessing vault. This does not look like a good security practice.