I’m trying to increase vault-agent-injector replicas to 2 in gke cluster but I am seeing an issue.
if I increase the replicas count to 2 here, deployment manifest sets a new variable called AGENT_INJECT_USE_LEADER_ELECTOR
and after deployment, when 2 new pods come, one of them goes to crashloopbackoff as it is not able to identify the leader. I tried a lot to find the root cause but we were not lucky enough.
PODS STATUSES -
POD NAME: vault-agent-injector-6bd5f47499-6lmmj
REPLICASET: vault-agent-injector-6bd5f47499
old pod with AGENT_INJECT_USE_LEADER_ELECTOR env variable available above replicaset and is going into crashloopbackoff
POD NAME: vault-agent-injector-bb7cbd88f-kzrf4
REPLICASET: vault-agent-injector-bb7cbd88f
old pod, ideally it should be terminated after helm upgrade but still running and in good health
POD NAME: vault-agent-injector-bb7cbd88f-z97zv
REPLICASET: ReplicaSet/vault-agent-injector-bb7cbd88f
new pod and running healthy
REPLICASETS -
operations vault-agent-injector-bb7cbd88f 2 2 2 40d sidecar-injector hashicorp/vault-k8s:1.1.0 app.kubernetes.io/instance=vault,app.kubernetes.io/name=vault-agent-injector,component=webhook,pod-template-hash=bb7cbd88f
operations vault-agent-injector-6bd5f47499 1 1 0 12d sidecar-injector hashicorp/vault-k8s:1.1.0 app.kubernetes.io/instance=vault,app.kubernetes.io/name=vault-agent-injector,component=webhook,pod-template-hash=6bd5f47499
ERRORED POD LOGS -
Using internal leader elector logic for webhook certificate management
Listening on ":8080"...
2023-02-21T15:23:38.676Z [INFO] handler: Starting handler..
2023-02-21T15:23:38.777Z [INFO] handler.certwatcher: Updated certificate bundle received. Updating certs...
2023-02-21T15:23:38.777Z [WARN] handler.certwatcher: Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...
2023-02-21T15:23:38.777Z [INFO] handler.certwatcher: Webhooks changed. Updating certs...
2023-02-21T15:23:38.777Z [WARN] handler.certwatcher: Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...
2023-02-21T15:23:38.777Z [INFO] handler.certwatcher: Webhooks changed. Updating certs...
2023-02-21T15:23:38.777Z [WARN] handler.certwatcher: Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...
2023-02-21T15:23:38.777Z [INFO] handler.certwatcher: Webhooks changed. Updating certs...
2023-02-21T15:23:38.777Z [WARN] handler.certwatcher: Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...
2023-02-21T15:23:38.777Z [INFO] handler.certwatcher: Webhooks changed. Updating certs...
2023-02-21T15:23:38.777Z [WARN] handler.certwatcher: Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...
2023-02-21T15:23:38.777Z [INFO] handler.certwatcher: Webhooks changed. Updating certs...
2023-02-21T15:23:38.777Z [WARN] handler.certwatcher: Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...
2023-02-21T15:23:38.777Z [INFO] handler.certwatcher: Webhooks changed. Updating certs...
2023-02-21T15:23:38.777Z [WARN] handler.certwatcher: Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...
2023-02-21T15:23:38.777Z [INFO] handler.certwatcher: Webhooks changed. Updating certs...
2023-02-21T15:23:38.777Z [WARN] handler.certwatcher: Could not load TLS keypair: tls: failed to find any PEM data in certificate input. Trying again...
I0221 15:23:39.719836 1 request.go:682] Waited for 1.045781175s due to client-side throttling, not priority and fairness, request: GET:https://10.132.8.1:443/apis/batch/v1?timeout=32s
2023-02-21T15:23:41.524Z [ERROR] handler: http: TLS handshake error from 10.132.18.1:60704: no certificate available
2023-02-21T15:23:41.524Z [ERROR] handler: http: TLS handshake error from 10.132.18.1:60694: no certificate available
2023-02-21T15:23:43.524Z [ERROR] handler: http: TLS handshake error from 10.132.18.1:60720: no certificate available
2023-02-21T15:23:43.525Z [ERROR] handler: http: TLS handshake error from 10.132.18.1:60730: no certificate available
2023-02-21T15:23:43.526Z [ERROR] handler: http: TLS handshake error from 10.132.18.1:60746: no certificate available