K8s injector not injecting sidecar container

Vault
1

Hello, I’m trying to get vault working with the injector on a GKE hosted cluster; this is the first time I’ve tried to use injection so it’s possible I’m just failing but from my research I haven’t found a solution yet.

My test application is deploying and starting but the secret container is not present in the pod (only one container shows) and when I look at the FS there is no mounted /vault path.

My test app manifest is as follows:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-test
  labels:
    app: vault-test
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vault-test
  labels:
    app: vault-test
spec:
  selector:
    matchLabels:
      app: vault-test
  replicas: 1
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/tls-skip-verify: "true"
        vault.hashicorp.com/agent-inject-secret-test: "secret/test"
        vault.hashicorp.com/agent-inject-template-test: |
          {{ with secret "secret/test" -}}
            export TEST_KEY="{{ .Data.testkey }}"
          {{-end }}
        vault.hashicorp.com/role: "test"
      labels:
        app: vault-test
    spec:
      serviceAccountName: vault-test
      containers:
        - name: app
          image: alpine:latest
          command: ["ls"]

You can see in the logs for the app where /vault does not exist:

kubectl logs vault-test-5f5d794f47-gn5t5 -n vault
bin
dev
etc
home
lib
media
mnt
opt
proc
root
run
sbin
srv
sys
tmp
usr
var

I’ve created the correct service account, policy and secret data in vault:

image
image1405×289 14.6 KB

image
image1041×920 42.3 KB

image
image1046×556 13.2 KB

I was looking at Injector does not inject sidecar container · Issue #32 · hashicorp/vault-k8s · GitHub because it appears to be a very similar issue but at this point I’m not sure it’s the same, for one thing I had port 8080 opened by my cluster admins and when that didn’t make a difference I was looking more closely and noticed the mutating webhook is set to use 443 anyway.

In any event, it looks like the injector is never getting a call from the webhook, the logs only show certwatcher updates such as:

2023-02-06T19:54:26.003Z [INFO] handler.certwatcher: Updated certificate bundle received. Updating certs…

nothing related to the deployment shows.

For knowledge, this is an HA deployment using Integrated Storage. Thanks for any thoughts.