Empty secret result from vault injector in k8s

I have vault server & agent setup in two k8s.
and running nice in dev env cluster, but does not in prod env cluster.

I have configured the same in both k8s clusters, and only prod vault generates empty secret file for pods. although, In dev env, it generates right secret file.
I want to here how to debug.

here’s my configuration.(both same in dev, prod env)

  • running vault server in HA mode(dev replica=1, prod replica=2)
  • installed via helm
  • all initialized and unsealed
  • k8s auth enabled, configured, bound to SA with right policy
  • pod deployments have annotations for vault injector
        vault.hashicorp.com/agent-inject: 'true'
        vault.hashicorp.com/agent-inject-secret-credential.json: nubison
        vault.hashicorp.com/agent-inject-template-credential.json: |
          {{ range sprig_list
          {{ $key := . | regexReplaceAll "^nubison/" "" -}}
          {{ $collect := (secret .).Data.data | explodeMap -}}
          {{ scratch.MapSet "vars" $key $collect -}}
          {{ end -}}
          {{ scratch.Get "vars" | toJSONPretty -}}
        vault.hashicorp.com/agent-run-as-user: '1337'
        vault.hashicorp.com/role: nubison-read

in dev cluster, agent generates credential.json

  "mysql": {
    "user": "lalala"
  "server:" {
    "token": "alalal"

in prod cluster, agent generates credential.json

  "mysql": {},
  "server:" {}

prod vault server has the secrets too.
and I see no error or suspicious agent log.

any advice please?

Turn the Vault audit logs on if you’re not using that feature already, and review the actual requests and responses being made.

1 Like

hi @maxb as you suggested, I enabled the audit log.
I’ve looked around the logs on both sides (dev/prod) and I see no differences.
The authentication performed well, and I can see the vault server responded secrets my agent asked for.
Still, the agent generates an empty credential.json :frowning:

Now at least I know auths and responses are made ok.

What can I do to dig deeper?

Thank you.

Sorry, I have no further suggestions. The problem most likely lies in some difference between your dev and prod that is not mentioned in this post.