I’m having a hard time trying to use Vault secrets inside a Pod. Taking this video as base: Injecting Vault Secrets Into Kubernetes Pods via a Sidecar - YouTube
I’m experiencing this issue Vault agent can't authenticate using k8s 1.21 · Issue #562 · hashicorp/vault-helm · GitHub, but as user tvoren commented, if even with disable_iss_validation=true, that would mean the problem is elsewhere.
Just for testing, I launched a new EKS cluster using 1.20 instead of 1.21, and can confirm the issue is still happening.
Here’s everything I’ve done:
$ helm install vault hashicorp/vault --set server.dev.enabled=true
$ vault auth enable kubernetes
$ vault write auth/kubernetes/config \
token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
kubernetes_host="https://${KUBERNETES_PORT_443_TCP_ADDR}:443" \
kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
$ vault write auth/kubernetes/role/app \
bound_service_account_names=app \
bound_service_account_namespaces=default \
policy=app
$ cat <<EOF > /home/vault/policy.hcl
path "secret*" {
capabilities = ["read"]
}
EOF
$ vault policy write app /home/vault/policy.hcl
$ vault kv put secret/app a=b
Then, on the Kubernetes side, I have a deployment with:
annotations:
vault.hashicorp.com/agent-inject: 'true'
vault.hashicorp.com/agent-inject-secret-app: 'secret/app'
vault.hashicorp.com/role: 'app'
....
.
..
...
serviceAccountName: app
and a service account
apiVersion: v1
kind: ServiceAccount
metadata:
name: app
The vault-agent-init container inside the pod is always returning
[WARN] (view) vault.read(secret/app): vault.read(secret/app): Error making API request.
URL: GET http://vault.default.svc:8200/v1/secret/app
Code: 403. Errors:
* 1 error occurred:
* permission denied
When using Kubernetes 1.21, the output is exactly as mentioned in the GitHub issue linked above, even after properly defining the issuer or just disabling the issuer at all with disable_iss_validation.