I have following annotations for my apps
extra_podAnnotations:
vault.hashicorp.com/agent-inject: ‘true’
vault.hashicorp.com/role: ‘demo-workflow-be-role’
vault.hashicorp.com/secret-volume-path: config
vault.hashicorp.com/agent-inject-secret-application.yaml:
vault.hashicorp.com/agent-inject-template-application.yaml: |
{{ with secret “” -}}
{{ .Data.data | toYAML }}
{{- end }}
This works perfectly most of the time. vault init container is injected properly and it all works fine. But sometimes, rarely the initContainer injection fails and the pods goes into CrashLoopbackOff state and main application container cant run without vault secrets.
But once we manually delete the pod, the init container is injected properly and pod will work fine.
I found that the sometimes vault-injector pods goes down due to resource issues or other issues and it is causing the init container injection to fail. So I increased the replica count of injector pod to 2. Even then, I am rarely facing the issue were pods are in crashloopbackoff due to initContainer injection failure. I don’t see any error logs in injector pod except for these error rarely in logs:
k -n hassim-vault logs hassim-agent-injector-f6fd68c8c-r5bk2 | grep -v “handler.certwatcher”
2023-09-11T12:31:06.846Z [ERROR] handler: http: TLS handshake error from 14.0.179.143:50654: remote error: tls: bad certificate
2023-09-11T12:39:39.109Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=30s
2023-09-11T12:39:39.314Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=30s
2023-09-11T12:39:39.609Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=30s
2023-09-11T12:39:39.801Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=30s
2023-09-11T12:39:39.817Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=30s
2023-09-11T12:39:39.895Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=30s
2023-09-11T12:39:40.039Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=30s
2023-09-11T12:39:42.208Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=30s
2023-09-11T12:39:42.509Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=30s
2023-09-11T12:40:04.807Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=30s
2023-09-11T12:41:16.319Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=30s
2023-09-11T12:50:15.111Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=30s
2023-09-11T12:50:15.120Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=30s
2023-09-11T12:50:15.123Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=30s
2023-09-11T12:50:15.126Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=30s
2023-09-11T12:50:15.128Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=30s
2023-09-11T12:50:16.086Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=30s
2023-09-11T12:50:16.086Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=30s
2023-09-11T12:50:17.593Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=30s
2023-09-11T12:50:18.700Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=30s
2023-09-11T12:50:27.586Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=30s
2023-09-11T12:50:31.747Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=30s
2023-09-11T12:50:31.776Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=30s
2023-09-11T12:50:31.802Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=30s