Vaults secrets injected by vault sidecar container inside the pod are visible to kubernetes cluster users/admin

I have integrated the external vault into kubernetes cluster. Vault is injecting the secrets into shared volume “/vault/secrets” inside the pod which can be consumed by application container. Till now everything looks good.

But I can see security risk by inserting the secrets into shared volume in plain text as anyone can access the application secrets who has access to the kubernetes cluster.

Example: Secrets are injected into shared volume at /vault/secrets/config

Now, If kubernetes cluster admin logged in and he can access the pod along with secrets available at the shared volume in plain text format.

Kubectl exec -it command will be used to enter into pod.

In this case, my concern is cluster admin can access the application secrets (Ex: database passwords) which is security risk. In my scenario vault admin is different and kubernetes cluster admin is different.

Please provide your inputs on this. Thanks in advance

Yes you need to be careful about who has access to your application as well as all the surrounding infrastructure, as people with such access would be able to obtain credentials. As you mentioned someone with admin privileges to Kubernetes would be able to look at secrets, but also would be able to exec into a pod or possibly dump memory (to find credentials).

To reduce the risk you need to have robust access control and audit processes to reduce the liklihood of inappropriate activities (as well as detecting them if they do occur).

The use of alternative auth options (such as the Kubernetes auth) wouldn’t really help as admins would still be able to intercept things and grant themselves access to credentials. You might be able to mitigate risks slightly by using single use wrapped tokens in some situations, although that would bring its own problems when pods recycle.

1 Like

I agree with @stuart-c. It’s one of the tradeoffs of having auto-auth and not having to integrated Vault into your application.
I’m no Kubernetes expert but I believe “cluster” wise, you’re safe from other containers/users as they wouldn’t be able to mount your PV, a PVC will lock your storage. As long as the admin can get into your pod though you’re still at some risk there. But that’s also true for almost any auth. If someone has access they can get to the secrets. The trick is to reduce that risk as low as possible.

@aram @stuart-c Thank you for your swift response.
Is it possible export the vault secrets as environment variable without writing to shared volume path? Just a thought.
My intention is to avoid a copy of the secret in a file.

Yes I have gone through this but still it is writing the secrets to /vault/secrets/ path

Environment variables are no better. Someone with permissions (e.g. a cluster admin) could easily exec into a pod and view all environment variables and their contents. If anything files are better than env vars.

Yes. I agree with you. Only option we have is, restrict the access of the users to cluster.
I am thinking of solution similar to this one, provided at below link
https://codeengineered.com/blog/2020/k8s-exec-access/

n your example, does it mean that

{{ with secret “secret/data/web” -}}
export api_key="{{ .Data.data.payments_api_key }}"
{{- end }}

export api_key creates the new env variable api_key?

Correct, already exported.

Doesn’t work for me. What’s wrong here?

spec:
template:
metadata:
annotations:
vault.hashicorp.com/agent-inject: ‘true’
vault.hashicorp.com/agent-inject-status: ‘update’
vault.hashicorp.com/role: ‘internal-app’
vault.hashicorp.com/agent-inject-secret-database-config.txt: ‘internal/data/database/config’
vault.hashicorp.com/agent-inject-template-database-config.txt: |
{{- with secret “internal/data/database/config” -}}
export api_key = postgresql://{{ .Data.data.username }}:{{ .Data.data.password }}@postgres:5432/wizard
{{- end -}}

This is the list of env vars:

Defaulted container “orgchart” out of: orgchart, vault-agent, vault-agent-init (init)
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=orgchart-55c76c489d-qsw6c
TERM=xterm
KUBERNETES_PORT_443_TCP_PROTO=tcp
VAULT_AGENT_INJECTOR_SVC_SERVICE_PORT=443
VAULT_PORT_8200_TCP_ADDR=10.106.25.116
VAULT_PORT_8201_TCP_PROTO=tcp
VAULT_PORT_8201_TCP_PORT=8201
KUBERNETES_SERVICE_HOST=10.96.0.1
KUBERNETES_SERVICE_PORT=443
VAULT_SERVICE_HOST=10.106.25.116
VAULT_AGENT_INJECTOR_SVC_PORT_443_TCP_ADDR=10.102.39.247
VAULT_PORT_8200_TCP_PROTO=tcp
VAULT_PORT_8201_TCP=tcp://10.106.25.116:8201
VAULT_AGENT_INJECTOR_SVC_PORT_443_TCP_PROTO=tcp
VAULT_SERVICE_PORT_HTTP=8200
KUBERNETES_PORT=tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
VAULT_AGENT_INJECTOR_SVC_SERVICE_PORT_HTTPS=443
VAULT_AGENT_INJECTOR_SVC_PORT=tcp://10.102.39.247:443
VAULT_AGENT_INJECTOR_SVC_PORT_443_TCP=tcp://10.102.39.247:443
VAULT_SERVICE_PORT_HTTPS_INTERNAL=8201
VAULT_PORT_8201_TCP_ADDR=10.106.25.116
KUBERNETES_SERVICE_PORT_HTTPS=443
VAULT_AGENT_INJECTOR_SVC_SERVICE_HOST=10.102.39.247
VAULT_PORT=tcp://10.106.25.116:8200
VAULT_PORT_8200_TCP_PORT=8200
VAULT_AGENT_INJECTOR_SVC_PORT_443_TCP_PORT=443
VAULT_SERVICE_PORT=8200
VAULT_PORT_8200_TCP=tcp://10.106.25.116:8200
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
HOME=/root