When I shut down and restart the nodes of a cluster (for example by removing and recreating the cluster node pool on GKE), I see that Pods are already being scheduled before the mutating webhook of the injector is created. This causes the init and/or sidecar container to be missing on these Pods, and it does not repair itself once the injector is online.
What would be a recommended approach for making sure we are ready to inject secrets before launching the annotated Pods that depend on it when booting a cluster with existing resources?
I actually solved it by setting the failure policy of the mutating webhook configuration to Fail instead of he default Ignore. This will stall creation of any annotated Pods until the admission controller is running. I guess we could make this configurable in the Helm chart.