HashiCorp Vault doesn't inject SideCar container to new pods

Vault
1

Our application runs on Kubernetes and secrets are basically stored in HashiCorp Vault. We are using the AppRole Authentication method where the Vault agent resides in the same cluster but, the Vault Server runs in a different Kubernetes cluster.

I’m passing the below annotations to my Kubernetes manifests but the issue I’m having is, when the Pod gets automatically ReScheduled (Due to autoscaling etc.) the Vault agent doesn’t inject the SideCar container to the newly rescheduled pod.

NOTE: This only happens when the Pod gets ReScheduled automatically.

annotations:
        vault.hashicorp.com/agent-inject: 'true'
        vault.hashicorp.com/agent-vault-addr: 'URL-Of-Vault-Server'
        vault.hashicorp.com/auth-type: 'approle'
        vault.hashicorp.com/auth-path: 'auth/approle'
        vault.hashicorp.com/auth-config-role-id-file-path: '/vault/custom/role-id'
        vault.hashicorp.com/auth-config-secret-id-file-path: '/vault/custom/secret-id'
        vault.hashicorp.com/agent-extra-secret: 'my-approle'
        vault.hashicorp.com/role: 'myrole'
        vault.hashicorp.com/auth-config-remove_secret_id_file_after_reading: 'false'
        vault.hashicorp.com/agent-inject-secret-config.env: 'kv/mysecrets/secrets'
        vault.hashicorp.com/agent-inject-template-config.env: |
          {{ with secret "kv/mysecrets/secrets/" -}}
             export JAVA_TOOL_OPTIONS="Some_Key={{ .Data.data.SECRET }} "
          {{- end }}

Any idea why it doesn’t inject the SideCar container to the newly rescheduled pods?

Do I need to pass these Annotations also?

vault.hashicorp.com/agent-inject-status
vault.hashicorp.com/agent-pre-populate
1 Like
2

Hi, did you get any solution to this?

3

Exactly the same problem here and only in init mode. And only for rescheduled POD too.

I try with the sidecar mode now to see if it change something.

4

@jmcollin78 What annotations you’ve passed in for sidecar mode? Can you share it?

5

@pragya5833 Nope. Still the same. So we are using a different agent now.

6

Hello, here is my config:

vault.hashicorp.com/agent-init-first: true
vault.hashicorp.com/agent-inject: true
vault.hashicorp.com/agent-inject-secret-action_xxxx.json: secret/xxxxx
vault.hashicorp.com/agent-inject-status: injected
vault.hashicorp.com/agent-inject-template-xxxx.json: {{ with secret "secret/xxx" }}...More
vault.hashicorp.com/agent-pre-populate-only: true
vault.hashicorp.com/role: xxxxxx
vault.hashicorp.com/tls-skip-verify: true

I’m trying with the sidecar mode:

oneagent.dynatrace.com/injected: true
vault.hashicorp.com/agent-inject: true
vault.hashicorp.com/agent-inject-status: injected
vault.hashicorp.com/agent-pre-populate: true
vault.hashicorp.com/agent-pre-populate-only: false
vault.hashicorp.com/role: xxxx
vault.hashicorp.com/tls-skip-verify: true

to see if it is a workaround. Seems good but I don’t hav yet the formal proof.

1 Like
7

See if this helps

Also try getting the Pod yaml definition and look for the container list in the Pod spec

From what you’re saying the admission webhook is not mutating the pod spec

Do you want to have the agent as both an init container and sidecar or just one of them?

8

Hell, thank you for your answer.

Initialy I wanted to have the vault injector in init mode only. But, sometimes it is not started when scheduler restarts my PODs.

With the Vault with both mode (init and sidecar) and make applicative container fails if secrets are not present , I do NOT reproduce the problem anymore.

This seems to be a valid workaround.