Intermittent TCP connection time out from K8s to external Vault

Hello All,

We have been using external vault with our K8s environment and using helm for vault injector package.

Up recently the setup was working fine and no changes performed vault infra.
We have 4 K8s clusters connecting to same vault, 3 of them working fine, but on 1 cluster often seen pods stuck in “Init 0/1” state intermittently.

Sometimes, restarting pods fixes the issue but comes back in same state after aws spot rebalancing.

Env Info:

K8s: AWS EKS 1.23
Vault: 1.5.4 (with consul) + AWS LB
Helm Vault Chart: hashicorp/vault 0.23.0
Note: (Same setup working fine in 3 other environments with vault chart version 0.8.0, this intermittent issue is also stopping us in upgrading our vault setup)

Vault Init Pod Error:
==> Vault agent started! Log data will stream in below:

==> Vault agent configuration:

                 Cgo: disabled
           Log Level: info

2023-02-06T13:07:07.568Z [INFO] sink.file: creating file sink
2023-02-06T13:07:07.568Z [INFO] sink.file: file sink configured: path=/home/vault/.token mode=-rw-r-----
Version: Vault v1.12.1, built 2022-10-27T12:32:05Z
Version Sha: e34f8a14fb7a88af4640b09f3ddbb5646b946d9c

2023-02-06T13:07:07.568Z [INFO] sink.server: starting sink server
2023-02-06T13:07:07.568Z [INFO] template.server: starting template server
2023-02-06T13:07:07.568Z [INFO] (runner) creating new runner (dry: false, once: false)
2023-02-06T13:07:07.568Z [INFO] auth.handler: starting auth handler
2023-02-06T13:07:07.568Z [WARN] (clients) disabling vault SSL verification
2023-02-06T13:07:07.568Z [INFO] auth.handler: authenticating
2023-02-06T13:07:07.568Z [WARN] (clients) disabling nomad SSL verification
2023-02-06T13:07:07.569Z [INFO] (runner) creating watcher
2023-02-06T13:07:37.570Z [ERROR] auth.handler: error authenticating: error=“Put "https://<external-vault-address>/login": dial tcp 10...:443: i/o timeout" backoff=1s
2023-02-06T13:07:38.571Z [INFO] auth.handler: authenticating
2023-02-06T13:08:08.571Z [ERROR] auth.handler: error authenticating: error="Put "https://<external-vault-address>/login": dial tcp 10.
..:443: i/o timeout” backoff=1.55s

Any Thoughts?

Did you find the root cause for this issue?