Can't get vault-agent-init container to get a secret from Vault

Vault
,
1

I’m having a hard time trying to use Vault secrets inside a Pod. Taking this video as base: Injecting Vault Secrets Into Kubernetes Pods via a Sidecar - YouTube

I’m experiencing this issue Vault agent can't authenticate using k8s 1.21 · Issue #562 · hashicorp/vault-helm · GitHub, but as user tvoren commented, if even with disable_iss_validation=true, that would mean the problem is elsewhere.

Just for testing, I launched a new EKS cluster using 1.20 instead of 1.21, and can confirm the issue is still happening.

Here’s everything I’ve done:

$ helm install vault hashicorp/vault --set server.dev.enabled=true

$ vault auth enable kubernetes

$ vault write auth/kubernetes/config \
   token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
   kubernetes_host="https://${KUBERNETES_PORT_443_TCP_ADDR}:443" \
   kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

$ vault write auth/kubernetes/role/app \
bound_service_account_names=app \
bound_service_account_namespaces=default \
policy=app

$ cat <<EOF > /home/vault/policy.hcl
path "secret*" {
  capabilities = ["read"]
}
EOF

$ vault policy write app /home/vault/policy.hcl

$ vault kv put secret/app a=b

Then, on the Kubernetes side, I have a deployment with:

annotations:
        vault.hashicorp.com/agent-inject: 'true'
        vault.hashicorp.com/agent-inject-secret-app: 'secret/app'
        vault.hashicorp.com/role: 'app'
....
.
..
...
serviceAccountName: app

and a service account

apiVersion: v1
kind: ServiceAccount
metadata:
  name: app

The vault-agent-init container inside the pod is always returning

[WARN] (view) vault.read(secret/app): vault.read(secret/app): Error making API request.

URL: GET http://vault.default.svc:8200/v1/secret/app
Code: 403. Errors:

* 1 error occurred:
        * permission denied

When using Kubernetes 1.21, the output is exactly as mentioned in the GitHub issue linked above, even after properly defining the issuer or just disabling the issuer at all with disable_iss_validation.

2

Fuck me, and fuck vault!

The error was in the role definition. The parameter is policies instead of policy, but it didn’t report back any error!