Vault Sidecar no activity despite annotations

Vault
1

Hello,

Short version of my problem - I have a setup for the Vault Sidecar Injector with all components healthy, and sidecar annotations specified in an example app that come from a previously working setup, but the injector is not responding to the annotations, and it is not logging anything meaningful.

I’ve got logLevel: "trace" specified in the Helm yaml installing the injector, and also in the annotations. When I look at logs with kubectl -n vault logs vault-agent-injector-867f5fdffd-86cn5, all I get is

2022-04-05T14:51:04.107Z [INFO]  handler: Starting handler..
Listening on ":8080"...
2022-04-05T14:51:04.108Z [INFO]  handler.auto-tls: Generated CA
2022-04-05T14:51:04.148Z [INFO]  handler.certwatcher: Updated certificate bundle received. Updating certs...

Any advice on how to get more out of the injector in terms of its response to Kubernetes events and reaction to annotations?

2

Apparently since I am not seeing any log activity, the webhook that should be invoked on pod CREATE and UPDATE isn’t getting invoked by Kubernetes. I wonder if there are logs somewhere in Kubernetes to show me a MutatingWebHook processing error.

3

Please post the actual annotations you’re using. There is a bug with “ERROR”, but “TRACE” should be working.

spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "internal-app"
        vault.hashicorp.com/log-level: "TRACE"

You can also check the log level after with:

$ kubectl describe deployment <app>
4

Thanks for the reply. What I learned yesterday, the injector webhook isn’t even getting called. It should be called for every pod create/update irrespective of the pod’s annotations, and that should result in log output in the injector

// Handle is the http.HandlerFunc implementation that actually handles the
// webhook request for admission control. This should be registered or
// served via an HTTP server.
func (h *Handler) Handle(w http.ResponseWriter, r *http.Request) {
	h.Log.Info("Request received", "Method", r.Method, "URL", r.URL)

I don’t see Request received at all. Again the injector was installed via Helm with this values.yaml:

injector:
  enabled: true
  externalVaultAddr: "https://${mcs_service_name}.${vault_app_namespace}.svc.clusterset.local:8200"
  authPath: "auth/gcp"
  logLevel: "trace"

I’m stuck without logs as to why the webhook doesn’t get called.

That said, the sample app does have the annotations you mention (and several others).

5

Discovered a firewall with default deny was governing control plane to pod connection. Resolved after having my IT team open port 8080.