Empty secret result from vault injector in k8s

Vault
1

I have vault server & agent setup in two k8s.
and running nice in dev env cluster, but does not in prod env cluster.

I have configured the same in both k8s clusters, and only prod vault generates empty secret file for pods. although, In dev env, it generates right secret file.
I want to here how to debug.

here’s my configuration.(both same in dev, prod env)

  • running vault server in HA mode(dev replica=1, prod replica=2)
  • installed via helm
  • all initialized and unsealed
  • k8s auth enabled, configured, bound to SA with right policy
  • pod deployments have annotations for vault injector
        vault.hashicorp.com/agent-inject: 'true'
        vault.hashicorp.com/agent-inject-secret-credential.json: nubison
        vault.hashicorp.com/agent-inject-template-credential.json: |
          {{ range sprig_list
            "nubison/server"
            "nubison/mysql"
          -}}
          {{ $key := . | regexReplaceAll "^nubison/" "" -}}
          {{ $collect := (secret .).Data.data | explodeMap -}}
          {{ scratch.MapSet "vars" $key $collect -}}
          {{ end -}}
          {{ scratch.Get "vars" | toJSONPretty -}}
        vault.hashicorp.com/agent-run-as-user: '1337'
        vault.hashicorp.com/role: nubison-read

in dev cluster, agent generates credential.json

{
  "mysql": {
    "user": "lalala"
    ...
  },
  "server:" {
    "token": "alalal"
   ...
  }
}

in prod cluster, agent generates credential.json

{
  "mysql": {},
  "server:" {}
}

prod vault server has the secrets too.
and I see no error or suspicious agent log.

any advice please?

2

Turn the Vault audit logs on if you’re not using that feature already, and review the actual requests and responses being made.

1 Like
3

hi @maxb as you suggested, I enabled the audit log.
I’ve looked around the logs on both sides (dev/prod) and I see no differences.
The authentication performed well, and I can see the vault server responded secrets my agent asked for.
Still, the agent generates an empty credential.json :frowning:

Now at least I know auths and responses are made ok.

What can I do to dig deeper?

Thank you.

4

Sorry, I have no further suggestions. The problem most likely lies in some difference between your dev and prod that is not mentioned in this post.