Hi,
I am trying to automatically roll out my pods after secrets have been updated in Vault. To achieve that goal I decided to use vault-agent-injector which is working fine and injecting the secrets as files into the pods.
Is there a way to trigger a rollout restart deploy/statefulset after updating the secret?
Thank you in advance!
I have figured out how to update the secrets through the vault-agent-injector once updated in the vault. Below are the steps I follow.
spec:
containers:
- name: vault-agent
image: "hashicorp/vault:1.15.2"
command: ['sh', '-c', 'vault agent -config=/opt/configs/config.hcl -log-level=error']
resources:
requests:
memory: 50Mi
cpu: 100m
limits:
memory: 150Mi
cpu: 300m
volumeMounts:
- name: vault-configs
mountPath: /opt/configs/
- name: shared-data
mountPath: /vault/secrets/
volumes:
- name: vault-configs
configMap:
name: vault-configs
- name: shared-data
emptyDir: {}
apiVersion: v1
kind: ConfigMap
metadata:
name: vault-configs
annotations:
reloader.stakater.com/match: "true"
data:
config.hcl: |
auto_auth {
method "kubernetes" {
mount_path = "auth/hulk"
config = {
role = "hulk-app"
token_file_path = "/var/run/secrets/kubernetes.io/serviceaccount/token"
}
}
sink "file" {
config = {
path = "/home/vault/.vault-token"
}
}
}
vault {
address = "http://192.168.2.227:8200"
}
template_config {
static_secret_render_interval = "2m"
exit_on_retry_failure = true
}
env_template "USERNAME" {
contents = "{{ with secret \"hulk/secret/data/config\" }}{{ .Data.username }}{{ end }}"
error_on_missing_key = true
}
env_template "PASSWORD" {
contents = "{{ with secret \"hulk/secret/data/config\" }}{{ .Data.password }}{{ end }}"
error_on_missing_key = true
}
exec {
command = [". /opt/configs/secret_rotate.sh"]
restart_on_secret_changes = "always"
restart_stop_signal = "SIGTERM"
}
cache {
use_auto_auth_token = false
}
exit_after_auth = false
PS: remember to replace/update auto_auth, vault, env_template blocks with your vault settings and credentials accordingly. Create a config-init.hcl with the same content used by vault-agent-init
In the above settings let’s focus on:
static_secret_render_interval: If specified, configures how often the Vault Agent Template should render non-leased secrets such as KV v2. This setting will not change how often Vault Agent Templating renders leased secrets
exec: The exec block executes a command when the template is rendered and the output has changed.
command: This is the optional command to run when the template is rendered. The command will only run if the resulting template changes. The command must return within 30s (configurable), and it must have a successful exit code. Vault Agent is not a replacement for a process monitor or init system
This is a simple shell script that writes the secrets into the pod filesystem. Add it to the above configMap.
secret_rotate.sh: |
#!/bin/sh
echo "${USERNAME}" > /vault/secrets/username
echo "${PASSWORD}" > /vault/secrets/password
echo "===>"
sleep $(expr 60 \* 2)
Putting all together and deploying.
apiVersion: v1
kind: ConfigMap
metadata:
name: vault-configs
annotations:
reloader.stakater.com/match: "true"
data:
config.hcl: |
auto_auth {
method "kubernetes" {
mount_path = "auth/hulk"
config = {
role = "hulk-app"
token_file_path = "/var/run/secrets/kubernetes.io/serviceaccount/token"
}
}
sink "file" {
config = {
path = "/home/vault/.vault-token"
}
}
}
vault {
address = "http://192.168.2.227:8200"
}
template_config {
static_secret_render_interval = "2m"
exit_on_retry_failure = true
}
env_template "USERNAME" {
contents = "{{ with secret \"hulk/secret/data/config\" }}{{ .Data.username }}{{ end }}"
error_on_missing_key = true
}
env_template "PASSWORD" {
contents = "{{ with secret \"hulk/secret/data/config\" }}{{ .Data.password }}{{ end }}"
error_on_missing_key = true
}
exec {
command = [". /opt/configs/secret_rotate.sh"]
restart_on_secret_changes = "always"
}
cache {
use_auto_auth_token = false
}
exit_after_auth = false
config-init.hcl: |
auto_auth {
method "kubernetes" {
mount_path = "auth/hulk"
config = {
role = "hulk-app"
token_file_path = "/var/run/secrets/kubernetes.io/serviceaccount/token"
}
}
sink "file" {
config = {
path = "/home/vault/.vault-token"
}
}
}
vault {
address = "http://192.168.2.227:8200"
}
template_config {
static_secret_render_interval = "2m"
exit_on_retry_failure = true
}
env_template "USERNAME" {
contents = "{{ with secret \"hulk/secret/data/config\" }}{{ .Data.username }}{{ end }}"
error_on_missing_key = true
}
env_template "PASSWORD" {
contents = "{{ with secret \"hulk/secret/data/config\" }}{{ .Data.password }}{{ end }}"
error_on_missing_key = true
}
exec {
command = [". /opt/configs/secret_rotate.sh"]
restart_on_secret_changes = "always"
}
cache {
use_auto_auth_token = false
}
exit_after_auth = false
secret_rotate.sh: |
#!/bin/sh
echo "${USERNAME}" > /vault/secrets/username
echo "${PASSWORD}" > /vault/secrets/password
echo "===>"
sleep $(expr 60 \* 2)
apiVersion: apps/v1
kind: Deployment
metadata:
name: app
annotations:
reloader.stakater.com/search: "true"
labels:
app: app
spec:
replicas: 1
selector:
matchLabels:
app: app
template:
metadata:
labels:
app: app
spec:
containers:
- name: vault-agent
image: "hashicorp/vault:1.15.2"
command: ['sh', '-c', 'while true; do vault agent -config=/opt/configs/config.hcl -log-level=info; done']
resources:
requests:
memory: 50Mi
cpu: 100m
limits:
memory: 150Mi
cpu: 300m
volumeMounts:
- name: vault-configs
mountPath: /opt/configs/
- name: shared-data
mountPath: /vault/secrets/
- name: app
image: nginx:stable-alpine3.17
ports:
- name: app
containerPort: 80
resources:
requests:
memory: 50Mi
cpu: 100m
limits:
memory: 150Mi
cpu: 300m
readinessProbe:
initialDelaySeconds: 40
periodSeconds: 40
failureThreshold: 1
exec:
command:
- sh
- -c
- '. /opt/configs/restart_app.sh'
livenessProbe:
initialDelaySeconds: 40
periodSeconds: 40
failureThreshold: 1
exec:
command:
- sh
- -c
- '. /opt/configs/restart_app.sh'
volumeMounts:
- name: vault-configs
mountPath: /opt/configs/
- name: shared-data
mountPath: /vault/secrets/
volumes:
- name: vault-configs
configMap:
name: vault-configs
- name: shared-data
emptyDir: {}
Then:
kubectl apply -f configmap.yaml -f deploy.yaml
NAME SECRETS AGE
serviceaccount/default 0 3d1h
serviceaccount/hulk-app 0 21h
NAME DATA AGE
configmap/kube-root-ca.crt 1 3d1h
configmap/vault-credentials 2 21h
configmap/vault-configs 4 21h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/app 1/1 1 1 21h
NAME READY STATUS RESTARTS AGE
pod/app-5fb7d5b948-wfbfm 2/2 Running 0 21h
---
apiVersion: batch/v1
kind: CronJob
metadata:
name: rotate-secret
spec:
schedule: "*/5 * * * *"
successfulJobsHistoryLimit: 3
jobTemplate:
spec:
template:
spec:
serviceAccountName: "hulk-app"
containers:
- name: vault
image: "hashicorp/vault:1.15.2"
command: ['sh', '-c', 'vault kv put hulk/secret/data/config username=postgres password=$(head -c 16 /dev/urandom|base64)']
envFrom:
- configMapRef:
name: vault-credentials
restartPolicy: Never
kubectl apply -f cronjob.yaml
Watch for password changes in the pod(s)
postgres
Opp7lvNPbbHwnpqf/x0qVg==
postgres
fMjsfueY8+jyMI7tiPfScw==
To automatically roll out your pods in Kubernetes after updating secrets in Vault, you can use a custom script or a Kubernetes operator. The script or operator should monitor for changes in Vault secrets and then trigger a rollout restart of the relevant Deployment or StatefulSet using the kubectl rollout restart command. This approach requires additional configuration and possibly custom coding but allows for automated updates in response to changes in Vault.