Yeah I know … tons of posts about kubernetes auth and the JWT flow. But they dont answer this particular question.
So I am running through the Mount Vault Secrets though CSI walkthrough, except I am using an external vault. In my case running in Docker but it could be anywhere.
Vault is installed as a client with just the csi stuff enabled, like this
helm install vault hashicorp/vault \
--set "server.enabled=false" \
--set "injector.enabled=false" \
--set "csi.enabled=true"
I set up kubernetes authentication with just the path to my kubernetes cluster and the ca.crt.
I set up the application example from the walkthrough and create the access policies and the test secret in Vault.
But when I start the application pod, mounting the secret, I get a 403 in the Vault agent logs.
I go and read up on the documentation on JWT flows and how vault checks the service account JWTs and I believe that I am using this flow here? ( Use the Vault client’s JWT as the reviewer JWT )
Then I create the clusterrole binding for the service account running the Vault agent pod, like this
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: vault-auth-delegator
subjects:
- kind: ServiceAccount
name: vault-csi-provider
namespace: default
roleRef:
kind: ClusterRole
name: system:auth-delegator
apiGroup: rbac.authorization.k8s.io
But … I get a 403.
Only after I change the rolebinding to grant system:auth-delegator access to the APPLICATION POD service account does it work and I can mount the secret.
Why on earth would the application pod service account need system:auth-delegator rights? And not the vault agent?
I dont understand that. That’s seriously bothering me here. Like … there goes my weekend and not even alcohol will get me stop thinking about this. Please help me understand this?