K8s Vault Agent Injector: Decoding secrets using the 'agent-inject-command' annotation

Vault
,
1

Hi,

I’m injecting a base64 encoded truststore file into my container and then using the ‘agent-inject-command’ annotation in an attempt to decode the secret and write it to a file. Here is a snipped of my k8s manifest:

vault.hashicorp.com/agent-inject-secret-truststore-jks: "secret/directory/truststore_jks"
vault.hashicorp.com/agent-inject-file-truststore-jks: b64.truststore.jks
vault.hashicorp.com/secret-volume-path-truststore-jks: /home
vault.hashicorp.com/agent-inject-command-truststore-jks-truststore-jks: /bin/bash -c "base64 -d /home/b64.truststore.jks > /home/truststore.jks"

The result is that the encoded version is injected to the file, but the command does not run successfully, thus the decoded version does not exist in the container. The logs for the ‘VAULT-AGENT-INIT’ container don’t report any problem:

command
command1115×130 7.45 KB

I have tried the same command with kubectl exec and it had the desired result.

If someone could let me know where I’m going wrong or give me another method of achieving a similar goal it would be greatly appreciated.

Kind Regards.

2

Hi @JamieMac96,

in the docs, it says:

vault.hashicorp.com/agent-inject-command - configures Vault Agent to run a command after the template has been rendered. To map a command to a specific secret, use the same unique secret name: vault.hashicorp.com/agent-inject-command-SECRET-NAME . For example, if a secret annotation vault.hashicorp.com/agent-inject-secret-foobar is configured, vault.hashicorp.com/agent-inject-command-foobar would map a command to that secret.

After reading this description, I would expect the annotation key vault.hashicorp.com/agent-inject-command-truststore-jks instead of vault.hashicorp.com/agent-inject-command-truststore-jks=truststore-jks. Maybe that’s the problem?

Best
Nick

3

Hi Nick,

Apologies that’s just a typo I made when modifying the snippet for readability. I’ve tested and been able to execute individual commands using this method, the problem only arises when redirecting the output of the decode operation to the new file.

1 Like
4

Hi @JamieMac96,

how did you verify the file wan’t written? Is the directory the truststore is written to mounted into the other containers?

5

Hi @Nick-Triller ,

To verify I exec’d into the pod that I was trying to write the truststore to and checked the home directory. I’m not sure whether that folder is mounted to the other containers (assuming you mean the vault agent init container from which the command is being run) but I have not added any additional configuration to do so.

6

Hi @JamieMac96,

please try adding a shared volume to the init container and your application container. The containers in a pod have separate filesystems. The init container including it’s filesystem are discarded once the init container finishes.

With a volume, the data in the volume is not discarded after the init container finishes and the same volume can be made available to the application container.

It should look something like this (taken from the Kubernetes docs):

apiVersion: v1
kind: Pod
metadata:
  name: init-demo
spec:
  containers:
  - name: nginx
    image: nginx
    ports:
    - containerPort: 80
    volumeMounts:
    - name: workdir
      mountPath: /usr/share/nginx/html
  # These containers are run during pod initialization
  initContainers:
  - name: install
    image: busybox
    command:
    - wget
    - "-O"
    - "/work-dir/index.html"
    - http://info.cern.ch
    volumeMounts:
    - name: workdir
      mountPath: "/work-dir"
  volumes:
  - name: workdir
    emptyDir: {}

Best
Nick

7

I just remembered you use the injector. I think the injector defines a shared volume by default, but it’s probably not the directory you use. You can check by taking a look at the pod’s manifest with “kubectl get pod $NAME -n $NS -o yaml”.

Edit: According to this page, the shared volume is /vault/secrets per default.

This means you should put your truststore into this directory instead of home.

8

@Nick-Triller

As far as I’m aware specifying the annotation secret-volume-path-truststore-jks: /home sets up a shared volume similar to the default one provided in /vault/secrets in the /home directory. Either way I tried the method you suggested and unfortunately it did not work.

However, I have found a method to achieve my goal through the use of the Vault Agent’s templating engine. From the docs:

Vault Agent's Template functionality allows Vault secrets to be rendered to files using Consul Template markup

This meant that I just had to add base64Decode to the template that I was injecting to decode the secret. So the relevant configuration to retrieve the decoded secret is as follows:

vault.hashicorp.com/agent-inject-secret-truststore-jks: "secret/path/to/secret/truststore_jks"
vault.hashicorp.com/agent-inject-file-truststore-jks: truststore.jks
vault.hashicorp.com/secret-volume-path-truststore-jks: /home
vault.hashicorp.com/agent-inject-template-truststore-jks: |
      {{- with secret "secret/path/to/secret" -}}
      {{ base64Decode .Data.data.truststore_jks }}
      {{-  end }}

Many thanks for helping me through this problem!

2 Likes