Hello, everyone. I am trying to get the vault-agent-injector working in my K8s cluster, and am seeing an issue where the mutating web hook does not seem to get triggered. Can anyone please give feedback on the best next steps? I have included all my steps taken below.
Vault Version: 1.13.2
- In vault create a secret engine named test-kv . In test-kv secret engine create one secret named secret1:
{
"key1": "value1"
}
- Enable the injector and debug logging in the helm chart and apply. Verify the helm chart values are correctly assigned by running
helm get values:
injector:
enabled: true
logLevel: debug
- In vault create a policy named service-policy with the following:
path "test-kv/data/*" {
capabilities = ["read"]
}
Verify the path is valid by running read test-kv/data/secret1 from the Vault Web CLI.
-
Create a K8s service account in K8s cluster:
kubectl create serviceaccount session-service-account -
Create a Vault role to bind the policy to the K8s service account:
#enables Kubernetes authentication
write sys/auth/kubernetes type=kubernetes
write auth/kubernetes/role/session-service-role \
bound_service_account_names=session-service-account \
bound_service_account_namespaces=test \
policies=service-policy \
ttl=24h
- Verify it got set correctly by running this in the Vault web CLI:
read auth/kubernetes/role/session-service-role
- Reviewed the vault-agent-injector pod configuration.
kubectl get mutatingwebhookconfigurations vault-agent-injector-cfg -o yaml
In the output we can see it is enabled to run for all namespaces:
namespaceSelector: {}
- Create a simple pod to see if it will trigger the web hook:
apiVersion: v1
kind: Pod
metadata:
name: test-pod-for-vault
namespace: test
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/role: "session-service-role"
vault.hashicorp.com/agent-inject-secret-secret1: "test-kv/data/secret1"
spec:
serviceAccountName: session-service-account
containers:
- name: ubuntu
image: ubuntu:latest
The pod comes up successfully, but nothing gets added to the pod showing the vault-agent-injector is working. I tried the following troubleshooting steps to see what is causing that:
Injector Logs
Even though debug logging is enabled, nothing is there in the vault-agent-injector pod logs and container logs, so it seems the web hook does not get triggered for some reason. The logs only display content like the following:
[INFO] handler.certwatcher: Webhooks changed. Updating certs...
Network Policy
Reviewed the network policy and ensured my Vault namespace accepts ingress traffic from my test namespace.
As a test, I tried to exec into my test pod above and do a curl request with the CA cert to my K8s cluster:
curl --header "X-Vault-Token: $TOKEN" $VAULT_ADDR/v1/sys/health
This is successful. For $VAULT_ADDR, I am using the hostname and not the internal K8s DNS path (vault..svc.cluster.local).
Check Kubernetes Auth
a. Perform a GET request to https://<my_host>/v1/auth/kubernetes/auth/session-service/role. In the output, I can see the following:
bound_service_account_namespaces: "test"
policies: "service-policy"
b. Perform a GET request to https://<my_host>/v1/auth/kubernetes/config. In the output I see the settings are there. To test if the kubernetes_ca_cert is valid, I tried running kubectl get pod while manually passing the certificate there. It is working, so the cert seems valid.
For kubernetes_host I am using https://kubernetes.default.svc.cluster.local.
Admission Controller Configuration
At this point I wondered if mutating web hooks were enabled at all for the cluster. In the master node, I reviewed the kube-apiserver yaml and confirmed --enable-admission-plugins has both MutatingAdmissionWebhook and ValidatingAdmissionWebhook there.
Thank you in advance.
It really helps us get a feeling for what the issue might be.
. Perhaps actual Kube API container logs would help. It seems to me youâre managing the cluster yourself so it shouldnât be hard to get those.