Vault Sidecar no activity despite annotations


Short version of my problem - I have a setup for the Vault Sidecar Injector with all components healthy, and sidecar annotations specified in an example app that come from a previously working setup, but the injector is not responding to the annotations, and it is not logging anything meaningful.

I’ve got logLevel: "trace" specified in the Helm yaml installing the injector, and also in the annotations. When I look at logs with kubectl -n vault logs vault-agent-injector-867f5fdffd-86cn5, all I get is

2022-04-05T14:51:04.107Z [INFO]  handler: Starting handler..
Listening on ":8080"...
2022-04-05T14:51:04.108Z [INFO] Generated CA
2022-04-05T14:51:04.148Z [INFO]  handler.certwatcher: Updated certificate bundle received. Updating certs...

Any advice on how to get more out of the injector in terms of its response to Kubernetes events and reaction to annotations?

Apparently since I am not seeing any log activity, the webhook that should be invoked on pod CREATE and UPDATE isn’t getting invoked by Kubernetes. I wonder if there are logs somewhere in Kubernetes to show me a MutatingWebHook processing error.

Please post the actual annotations you’re using. There is a bug with “ERROR”, but “TRACE” should be working.

      annotations: "true" "internal-app" "TRACE"

You can also check the log level after with:

$ kubectl describe deployment <app>

Thanks for the reply. What I learned yesterday, the injector webhook isn’t even getting called. It should be called for every pod create/update irrespective of the pod’s annotations, and that should result in log output in the injector

// Handle is the http.HandlerFunc implementation that actually handles the
// webhook request for admission control. This should be registered or
// served via an HTTP server.
func (h *Handler) Handle(w http.ResponseWriter, r *http.Request) {
	h.Log.Info("Request received", "Method", r.Method, "URL", r.URL)

I don’t see Request received at all. Again the injector was installed via Helm with this values.yaml:

  enabled: true
  externalVaultAddr: "https://${mcs_service_name}.${vault_app_namespace}.svc.clusterset.local:8200"
  authPath: "auth/gcp"
  logLevel: "trace"

I’m stuck without logs as to why the webhook doesn’t get called.

That said, the sample app does have the annotations you mention (and several others).

Discovered a firewall with default deny was governing control plane to pod connection. Resolved after having my IT team open port 8080.