I have several kubernetes clusters on different provider and decided to try out the DigitalOcean k8s offering.
I launched a new cluster. To test things out I deployed a pod that uses the kubernetes auto_auth with the vault agent.
The vault agent works great in all the other clusters, but in this new cluster it gets stuck with this message:
auth.handler: error authenticating: error="Error making API request.
URL: PUT https://vault.domain:8200/v1/auth/k8s-cluster-id/login
Code: 403. Errors:
* permission denied
The pod has the token and certificate in the /var/run/secrets/kubernetes.io/serviceaccount/ directory as expected.
When I was configuring the cluster I wrote the auth config to the vault cluster like this:
vault write auth/${K8S_NAME}/config
token_reviewer_jwt="${SA_JWT_TOKEN}"
kubernetes_host="${K8S_HOST}"
kubernetes_ca_cert="${SA_CA_CRT}"
That reported success.
Vault has the auth role and policies that the service account needs.
At this point, I am really not sure what to check next. I just need someone to point me to the next troubleshooting step(s).
The solution is boring. After much learning, a nice side effect of chasing bugs, I noticed in the logs a statement about how the certificate of the tokenreviewer (k8s cluster) did not include support for the address I had given to vault. I updated that address with one listed in the certificate, and now all is well.
It would be nice if there were some more details on the login error than just. permission denied.
Hi @SunSparc, I’m facing the exact same issue - could you please describe how you solved it? I’m using a standalone K8s cluster on AWS EKS against an external Vault.
I am facing the same issue, I got the token and certificate from service account.
my is openshift, for me the below command not worked.
KUBE_CA_CERT=$(kubectl config view --raw --minify --flatten --output=‘jsonpath={.clusters.cluster.certificate-authority-data}’ | base64 --decode)
@SunSparc what help exactly to fix this issue.
@chinmaychandak, @saigopi, sorry to respond so late. My solution was this:
In other words, I was sending the auth request to a domain name (https://vault.domain) that was not in the certificate. When I updated to a domain name that was in the certificate (https://valid.vault.domain), the problem was solved.
I could have also solved the problem by creating a new certificate that included the domain name (https://vault.domain) that I was using.
Hopefully that helps someone.