Vault Injector Not Authorized in Vault-Agent-Init Container Logs

Vault
,
1

I am using the Vault Agent Injector in my K8s clusters. It’s working well in all with the same configuration that I apply using Terraform except for 1 where the vault agent receives an authentication error:

2023-08-08T18:27:41.464Z [INFO]  agent.auth.handler: authenticating 2023-08-08T18:27:41.476Z [ERROR] agent.auth.handler: error authenticating: error= | Error making API request. | | URL: PUT https://vault.vault.svc:8200/v1/auth/kubernetes/login | Code: 403. Errors: | | * permission denied backoff=3m46.16s

Pod yaml:

apiVersion: v1
kind: Pod
metadata:
  name: test-pod-for-vault
  namespace: shop
  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/role: "shop-vault-injector"
    vault.hashicorp.com/agent-inject-secret-secret1: "shop-kv/data/secret1"
    vault.hashicorp.com/tls-skip-verify: "true"
spec:
  serviceAccountName: vaultinjector-serviceaccount
  containers:
  - name: ubuntu
    image: ubuntu:latest
    command: ["bash"]
    args: ["-c", "sleep infinity"]

Auth method role:

name: shop-vault-injector
bound sa name: vaultinjector-serviceaccount
bound sa namespace: shop
generated token's policies: shop-vaultinjector-policy

And here’s the policy:

path "*" {
  capabilities = ["read"]
}

(I have changed the path to * to see if that was the issue, but it did not make a difference.)

Troubleshooting Steps Taken:

I wanted to see if my auth method was configured correctly. I use a custom kubernetes auth method for the vault injector, and then all my roles for the injector are under that.

As a test, I got the token for vaultinjector-serviceaccount in the shop namespace and tried to see if I could login:

kubectl get secret <service-account-secret> -o jsonpath="{.data.token}" | base64 --decode

vault write auth/vaultinjector-kubernetes/login role=shop-vault-injector jwt=<insert_token_from_above_command>

# This works, so I then try to list the contents of the secret from my local:

vault kv get shop-kv/secret1

# Secret contents are returned.

Wouldn’t we expect this to fail if the Vault Injector is getting the 403 error?

I have also confirmed the vault service account in my vault namespace has the needed cluster role bindings:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vault-auth-delegator-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: vault
  namespace: vault
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: vault-token-review
rules:
- apiGroups: [""]
  resources: ["serviceaccounts"]
  verbs: ["get"]
- apiGroups: ["authentication.k8s.io"]
  resources: ["tokenreviews"]
  verbs: ["create"]
- apiGroups:
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vault-token-review-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: vault-token-review
subjects:
- kind: ServiceAccount
  name: vault
  namespace: vault

Is there anything else I can try to troubleshoot the error further? I do not see any errors in my vault injector and vault pod logs.