[vault agent injector] ACL policy wildcard

Vault
1

Hi :slight_smile: ,

I am having an issue while fetching secrets from an external HA Vault cluster.

I have created a secret like below:

$ vault secrets enable -path=demo kv-v2
$ vault kv put demo/dummy DEMO_SECRET="this is secret"

To allow a k8s cluster to authenticate to Vault:

$ vault auth enable -path=demo kubernetes

$ vault policy write vault-agent-injector - <<EOF
path "secret/metadata/demo/*" {
  capabilities = ["read", "list"]
}

path "secret/data/demo/*" {
  capabilities = ["read", "list"]
}
EOF

$ vault write auth/demo/config \
        token_reviewer_jwt="$TOKEN_REVIEW_JWT" \
        kubernetes_host="$KUBE_HOST" \
        kubernetes_ca_cert="$KUBE_CA_CERT" \
        issuer="https://kubernetes.default.svc.cluster.local"

$ vault write auth/demo/role/vault-agent-injector \
        bound_service_account_names=vault-agent-injector \
        bound_service_account_namespaces='*' \
        policies=vault-agent-injector \
        ttl=24h

I am deploying a pod annotated accordingly for being able to fetch my secret ‘demo/dummy’:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo
  namespace: vault
spec:
  replicas: 1
  selector:
    matchLabels:
      app: demo
  template:
    metadata:
      labels:
        app: demo
      annotations:
        vault.hashicorp.com/agent-inject: 'true'
        vault.hashicorp.com/role: 'vault-agent-injector'
        vault.hashicorp.com/agent-inject-secret-env: 'secret/demo'
        vault.hashicorp.com/agent-inject-template-env: |
          {{ with secret "secret/demo/dummy" }}
            {{ range $key, $value := .Data.data }}
              export {{ $key }}={{ $value }}
            {{- end }}
          {{- end }}
    spec:
      serviceAccountName: vault-agent-injector
      containers:
        - name: demo
          image: alpine:latest
          imagePullPolicy: IfNotPresent
          command: ["/bin/sh",  "-c", "source /vault/secrets/env && sleep infinity"]

However, here is the logs of the sidecar container when deploying my demo pod:

==> Vault Agent started! Log data will stream in below:

2023-04-11T17:24:45.816Z [INFO]  agent.sink.file: creating file sink
2023-04-11T17:24:45.816Z [INFO]  agent.sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r-----
==> Vault Agent configuration:

           Api Address 1: http://bufconn
                     Cgo: disabled
               Log Level: info
                 Version: Vault v1.13.1, built 2023-03-23T12:51:35Z
             Version Sha: 4472e4a3fbcc984b7e3dc48f5a8283f3efe6f282

2023-04-11T17:24:45.817Z [INFO]  agent.template.server: starting template server
2023-04-11T17:24:45.817Z [INFO] (runner) creating new runner (dry: false, once: false)
2023-04-11T17:24:45.818Z [INFO]  agent.auth.handler: starting auth handler
2023-04-11T17:24:45.818Z [INFO]  agent.auth.handler: authenticating
2023-04-11T17:24:45.818Z [INFO] (runner) creating watcher
2023-04-11T17:24:45.818Z [INFO]  agent.sink.server: starting sink server
2023-04-11T17:24:45.848Z [INFO]  agent.auth.handler: authentication successful, sending token to sinks
2023-04-11T17:24:45.848Z [INFO]  agent.auth.handler: starting renewal process
2023-04-11T17:24:45.848Z [INFO]  agent.sink.file: token written: path=/home/vault/.vault-token
2023-04-11T17:24:45.849Z [INFO]  agent.sink.server: sink server stopped
2023-04-11T17:24:45.849Z [INFO]  agent: sinks finished, exiting
2023-04-11T17:24:45.848Z [INFO]  agent.template.server: template server received new token
2023-04-11T17:24:45.849Z [INFO] (runner) stopping
2023-04-11T17:24:45.849Z [INFO] (runner) creating new runner (dry: false, once: false)
2023-04-11T17:24:45.849Z [INFO] (runner) creating watcher
2023-04-11T17:24:45.849Z [INFO] (runner) starting
2023-04-11T17:24:45.853Z [WARN] vault.read(secret/demo/dummy): failed to check if secret/demo/dummy is KVv2, assume not: Error making API request.

URL: GET http://34.91.191.201:8200/v1/sys/internal/ui/mounts/secret/demo/dummy
Code: 403. Errors:

* preflight capability check returned 403, please ensure client's policies grant access to path "secret/demo/dummy/"
2023-04-11T17:24:45.855Z [WARN] (view) vault.read(secret/demo/dummy): vault.read(secret/demo/dummy): Error making API request.

URL: GET http://34.91.191.201:8200/v1/secret/demo/dummy
Code: 403. Errors:

* 1 error occurred:
	* permission denied

The authentication seems successful but there are two things I do not understand.
First of all I must be missing something because I have this warning:

[WARN] vault.read(secret/demo/dummy): failed to check if secret/demo/dummy is KVv2, assume not: Error making API request.

Last but not the least, logs say the return code was 403 due to permission denied and ask me to ensure the client’s policy grant access to path “secret/demo/dummy/”.

Why is there a trailing slash ?

According the documentation when creating ACL policies “/data” and “/metadata” should be prepended to my path for the KV v2 engine, and the wildcard should be the last character. This way all secrets under path ‘demo’ should be accessible.

What am I missing or doing wrong with my ACL policy ?

Any help will be greatly appreciated :slight_smile:

Best,

2

You have enabled a KVv2 secrets engine at path demo.

But you have written policies for a KVv2 secrets engine at path secret, with a path of demo inside the secrets engine.

You need to consistently decide what you are going to call your secrets engine and not pick different names for it in different places in the setup.

In particular, note that the path segment secret has no special meaning to Vault, but is sometimes used in the Vault documentation as the path of an arbitrary secrets engine - and this often turns out to be confusing to users.

The presence of the final trailing slash in this error message should probably be considered a bug. It seems unhelpfully misleading to me.

The “preflight capability check” referred to, is the templating engine performing a probe to determine whether the path you have specified is located within a KVv2 secrets engine, in which case it will automatically insert a /data/ path segment into the path at the appropriate position.

Ultimately it seems like all the problems have flowed from not enabling the KVv2 secrets engine at the same path as you go on to refer to it via in the rest of your setup.

3

Wonderful, thank you very much @maxb ! I should have been more careful while reading the doc and not being fooled by the ‘secret’ path. Truly sorry for the waste of time, I feel dumb.

Thanks again.

Best,