Permission denied when trying to read data from vault

[dil-kpogany]

Problem:
I try to connect our external vault to kubernetes so we could consume data from the external vault in the pods. When I try to start the application with the vault side-car container it stucks in Init:0/1 status. Log says the following:

==> Vault agent started! Log data will stream in below:

==> Vault agent configuration:

                     Cgo: disabled
               Log Level: info
                 Version: Vault v1.7.3
             Version Sha: 5d517c864c8f10385bf65627891

2021-07-23T18:09:14.910Z [INFO]  sink.file: creating file sink
2021-07-23T18:09:14.910Z [INFO]  sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r-----
2021-07-23T18:09:14.910Z [INFO]  template.server: starting template server
2021-07-23T18:09:14.910Z [INFO]  auth.handler: starting auth handler
[INFO] (runner) creating new runner (dry: false, once: false)
2021-07-23T18:09:14.910Z [INFO]  auth.handler: authenticating
[INFO] (runner) creating watcher
2021-07-23T18:09:14.911Z [INFO]  sink.server: starting sink server
2021-07-23T18:09:15.712Z [INFO]  auth.handler: authentication successful, sending token to sinks
2021-07-23T18:09:15.712Z [INFO]  auth.handler: starting renewal process
2021-07-23T18:09:15.713Z [INFO]  sink.file: token written: path=/home/vault/.vault-token
2021-07-23T18:09:15.713Z [INFO]  sink.server: sink server stopped
2021-07-23T18:09:15.713Z [INFO]  sinks finished, exiting
2021-07-23T18:09:15.713Z [INFO]  template.server: template server received new token
[INFO] (runner) stopping
[INFO] (runner) creating new runner (dry: false, once: false)
[INFO] (runner) creating watcher
[INFO] (runner) starting
2021-07-23T18:09:15.820Z [INFO]  auth.handler: renewed auth token
[WARN] (view) vault.read(secret/devwebapp): vault.read(secret/devwebapp): Error making API request.

URL: GET https://vault-primary.dl.dev.local/v1/secret/devwebapp
Code: 403. Errors:

* 1 error occurred:
        * permission denied

 (retry attempt 1 after "250ms")
[WARN] (view) vault.read(secret/devwebapp): vault.read(secret/devwebapp): Error making API request.

URL: GET https://vault-primary.dl.dev.local/v1/secret/devwebapp
Code: 403. Errors:

* 1 error occurred:
        * permission denied

 (retry attempt 2 after "500ms")
[WARN] (view) vault.read(secret/devwebapp): vault.read(secret/devwebapp): Error making API request.

URL: GET https://vault-primary.dl.dev.local/v1/secret/devwebapp
Code: 403. Errors:

* 1 error occurred:
        * permission denied

 (retry attempt 3 after "1s")
[WARN] (view) vault.read(secret/devwebapp): vault.read(secret/devwebapp): Error making API request.

URL: GET https://vault-primary.dl.dev.local/v1/secret/devwebapp
Code: 403. Errors:

* 1 error occurred:
        * permission denied

 (retry attempt 4 after "2s")

Steps I followed:
1. created the injector:

2. configured vault auth:

vault write auth/kubernetes/config \
        token_reviewer_jwt="$TOKEN_REVIEW_JWT" \
        kubernetes_host="$KUBE_HOST" \
        kubernetes_ca_cert="$KUBE_CA_CERT"

3. created policy in vault:

vault policy write devwebapp - <<EOF
path "secret/devwebapp/" {
  capabilities = ["read"]
}
EOF

4. Added secret to vault

vault secrets enable -path=secret kv
vault kv put /secret/devwebapp username="test_name" password="test_password"

5. Created role in vault

  vault write auth/kubernetes/role/devweb-app \
        bound_service_account_names=vault-auth \
        bound_service_account_namespaces=vault \
        policies=devwebapp \
        ttl=24h

6. Created a service account

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-auth
  namespace: vault
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
  namespace: vault
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
  - kind: ServiceAccount
    name: vault-auth
    namespace: vault

7. Created secret in the namespace to store certs

apiVersion: v1
data:
 ca-bundle.crt: LS0tLS1LSCRBDRFUR....
kind: Secret
metadata:
 name: root-canp-cert
type: Opaque

8. Created yaml for the app pod

---
apiVersion: v1
kind: Pod
metadata:
  name: vault-app
  labels:
    app: devwebapp-annotations
  annotations:
    vault.hashicorp.com/agent-inject: 'true'
    vault.hashicorp.com/role: 'devweb-app'
    vault.hashicorp.com/agent-inject-secret-credentials.txt: 'secret/devwebapp/'
    vault.hashicorp.com/tls-secret: 'root-canp-cert'
    vault.hashicorp.com/ca-cert: 'vault/tls/ca-bundle.crt'
spec:
  serviceAccountName: vault-auth
  containers:
  - name: vault-app
    image: nginx

[jeffsanicola]

Looks like you may be using KVv2

Try changing your policy to this:

vault policy write devwebapp - <<EOF
path "secret/data/devwebapp" {
  capabilities = ["read"]
}
EOF

Edit: If you’re using KVv1 then remove the trailing / from your policy path.

[jeffsanicola]

I just noticed that
vault.hashicorp.com/agent-inject-secret-credentials.txt: 'secret/devwebapp/'
has a trailing / as well.

Try removing that as well as the one from your KVv1 policy, if you didn’t already try that.

I’m still fairly new to the Kube injector configs so I’m not sure if anything is incorrect there but it looks like a policy/path issue to me based on the 403 error (if it was an authentication issue I would expect to see an error against the /auth/kubernetes/login path.

[dil-kpogany]

I just changed the policy to path “*” {
capabilities = [“read”]
}
This way it works, so it must be some path issue.

[Prashanth-M]

@dil-kpogany had similar issue, but in my case it was issue with k8s version. Keep an eye on that.
More info:

External vault init container stuck at Init:0/1 with context deadline exceeded error

Problem: I’m using external vault server which is hosted on seperate k8s cluster. Using pod annotations in my application pod trying to render the secrets. As mentioned in: [Integrate a Kubernetes Cluster with an External Vault | Vault - HashiCorp Learn] Injector created with external vault helm install vault hashicorp/vault \ --set "injector.externalVaultAddr=https:myvault.com" vault secret VAULT_HELM_SECRET_NAME=$(kubectl get secrets --output=json | jq -r '.items[].metadata | select(.n…

[dil-kpogany]

This is resolved now, it was really the issue of the trailing /

[fbucci1]

Thanks a lot for the detailed post! I was lost and the * gave me a temporary solution, allowing me to continue.
The actual path has to be “secret/data/devwebapp”, just like the path you use in the API to access the secret.

[Berg-it]

you saved my day: we must remove “*” from the path of policy in order to access to it

[Mo0rBy]

I found this in the Vault docs here

# Permit reading only "secret/foo". an attached token cannot read "secret/food"
# or "secret/foo/bar".
path "secret/foo" {
  capabilities = ["read"]
}

# Permit reading everything under "secret/bar". an attached token could read
# "secret/bar/zip", "secret/bar/zip/zap", but not "secret/bars/zip".
path "secret/bar/*" {
  capabilities = ["read"]
}