Integrate a kubernetes cluster with external vault tutorial broken?

Vault
, , ,
1

I’m following the tutorial here: Integrate a Kubernetes Cluster with an External Vault | Vault - HashiCorp Learn The purpose of this tutorial is to integrate a kubernetes cluster with an external vault. I’m following the tutorial exactly, except for the fact that I’m using a k8s cluster hosted in Azure, managed by Rancher, and I’m using a standalone vault server that’s not in dev mode. Everything works fine until I get to the point where I’m attempting to create the devwebapp-with-annotations pod. This pod leverages the vault-agent-injector pod to inject secrets from vault into the kubernetes pod. The pod creates, but it never comes up. Looking at the vault server logs, I’m getting:

Aug 03 22:15:29 vault-node-1 vault[691]: 2021-08-03T22:15:29.420Z [ERROR] auth.kubernetes.auth_kubernetes_389f4a26: login unauthorized due to: lookup failed: service account unauthorized; this could mean it has been deleted or recreated with a new token

The vault-agent-init sidecar container in the devwebapp-with-annotations pod is showing:

==> Vault agent started! Log data will stream in below:
==> Vault agent configuration:
                     Cgo: disabled
               Log Level: info
                 Version: Vault v1.8.0
             Version Sha: 82a99f14eb6133f99a975e653d4dac21c17505c7
2021-08-03T22:11:55.538Z [INFO]  sink.file: creating file sink
2021-08-03T22:11:55.539Z [INFO]  sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r-----
2021-08-03T22:11:55.539Z [INFO]  template.server: starting template server
2021-08-03T22:11:55.539Z [INFO] (runner) creating new runner (dry: false, once: false)
2021-08-03T22:11:55.539Z [INFO]  sink.server: starting sink server
2021-08-03T22:11:55.539Z [INFO]  auth.handler: starting auth handler
2021-08-03T22:11:55.539Z [INFO]  auth.handler: authenticating
2021-08-03T22:11:55.539Z [INFO] (runner) creating watcher
2021-08-03T22:11:55.595Z [ERROR] auth.handler: error authenticating:
  error=
  | Error making API request.
  |
  | URL: PUT http://external-vault:8200/v1/auth/kubernetes/login
  | Code: 403. Errors:
  |
  | * permission denied
   backoff=1s

The auth.handler error then repeats a bunch of times.

Am I missing something because I’m running this in a “real” environment or is the tutorial broken?

1 Like
2

Hey! This is a problem that’s caused by a change in the jwt format in k8s 1.21. You can read more about it and the solution here: Support new JWT token structure on k8s 1.21 · Issue #11953 · hashicorp/vault · GitHub

Quick fix:

kubectl proxy &

K8S_ISSUER=curl --silent http://127.0.0.1:8001/api/v1/namespaces/default/serviceaccounts/default/token \
  -H "Content-Type: application/json" \
  -X POST \
  -d '{"apiVersion": "authentication.k8s.io/v1", "kind": "TokenRequest"}' \
  | jq -r '.status.token' \
  | cut -d. -f2 \
  | base64 -D

vault write auth/kubernetes/config \
  token_reviewer_jwt=<YOUR_JWT> \
  kubernetes_host=<YOUR_HOST> \
  kubernetes_ca_cert=<YOUR_CERT> \
  issuer="$K8S_ISSUER"
1 Like
3

Is this something that will affect this tutorial too?

Because I also can’t get this to work.

4

Hi! Thanks for your response. Sorry I should have specified this in the original post, but this particular cluster is running v1.19.12 of k8s and Vault 1.7.3. Also I noticed the minikube tutorial is running 1.20.0. Would this still be a viable workaround or is it only for 1.21+ clusters?

5

Huh, it’s probably something else then. From the error log, it seems to just be some issue with your setup with k8s auth. Maybe the role registration wasn’t proper, or you misspelled the role/policy somewhere?