Hi,
i have deployed vault-agent using helm chart GitHub - hashicorp/vault-helm: Helm chart to install Vault and other associated components.. Agent pod’s liveness probe getting failed intermittently thus causing pod to restart / crash. Debug logs doesn’t has any relevant error that explains probe failure, however k8s events does show an error remote error: tls: internal error. Or net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
K8s Events
Warning Unhealthy pod/vault-agent-injector-868f4cfd48-7qv4p Liveness probe failed: Get "https://X.X.X.X:8080/health/ready": remote error: tls: internal error
Warning Unhealthy pod/vault-agent-injector-868f4cfd48-7qv4p Readiness probe failed: Get "https://X.X.X.X:8080/health/ready": remote error: tls: internal error
Normal Killing pod/vault-agent-injector-868f4cfd48-7qv4p Stopping container sidecar-injector
Warning Unhealthy pod/vault-agent-injector-868f4cfd48-7qv4p Liveness probe failed: Get "https://X.X.X.X:8080/health/ready": dial tcp X.X.X.X:8080: connect: no route to host
Warning Unhealthy pod/vault-agent-injector-868f4cfd48-7qv4p Readiness probe failed: Get "https://X.X.X.X:8080/health/ready": dial tcp X.X.X.X:8080: connect: no route to host
Warning Unhealthy pod/vault-agent-injector-5f7f9575cc-kp447 Liveness probe failed: Get "https://X.X.X.X:8080/health/ready": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Normal Killing pod/vault-agent-injector-5f7f9575cc-kp447 Container sidecar-injector failed liveness probe, will be restarted
Warning Unhealthy pod/vault-agent-injector-5f7f9575cc-kp447 Readiness probe failed: Get "https://X.X.X.X:8080/health/ready": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Vault Agent pod (leader) logs
2022-12-13 19:24:30 Error updating MutatingWebhookConfiguration: Unauthorized
2022-12-13 19:24:29 Error updating MutatingWebhookConfiguration: Unauthorized
2022-12-13 19:24:28 Error updating MutatingWebhookConfiguration: Unauthorized
Vault Agent pod (follower) logs
2022-12-13 19:24:19 Error listening: http: Server closed
2022-12-13 19:24:19 handler: Shutting down due to error: error=Unauthorized
2022-12-13 19:24:19 [ERROR] handler: Trouble becoming leader: error=Unauthorized
2022-12-13 19:24:18 [DEBUG] handler.auto-tls: Currently a follower
Environment / Versions details
K8s version = 1.21.14
Vault helm chart version = 0.16.1
Vault server version = 1.8.3 (deployed via helm chart)
Vault agent version = 0.13.1
Can someone please help troubleshooting this issue?