Vault agent unable to authenticate in new Kubernetes cluster

I have several kubernetes clusters on different provider and decided to try out the DigitalOcean k8s offering.

I launched a new cluster. To test things out I deployed a pod that uses the kubernetes auto_auth with the vault agent.

The vault agent works great in all the other clusters, but in this new cluster it gets stuck with this message:

auth.handler: error authenticating: error="Error making API request.

URL: PUT https://vault.domain:8200/v1/auth/k8s-cluster-id/login
Code: 403. Errors:

* permission denied

The pod has the token and certificate in the /var/run/secrets/kubernetes.io/serviceaccount/ directory as expected.

When I was configuring the cluster I wrote the auth config to the vault cluster like this:

vault write auth/${K8S_NAME}/config
    token_reviewer_jwt="${SA_JWT_TOKEN}"
    kubernetes_host="${K8S_HOST}"
    kubernetes_ca_cert="${SA_CA_CRT}"

That reported success.

Vault has the auth role and policies that the service account needs.

At this point, I am really not sure what to check next. I just need someone to point me to the next troubleshooting step(s).

The solution is boring. After much learning, a nice side effect of chasing bugs, I noticed in the logs a statement about how the certificate of the tokenreviewer (k8s cluster) did not include support for the address I had given to vault. I updated that address with one listed in the certificate, and now all is well.

It would be nice if there were some more details on the login error than just. permission denied.