External vault init container stuck at Init:0/1 with context deadline exceeded error

Problem: I’m using external vault server which is hosted on seperate k8s cluster. Using pod annotations in my application pod trying to render the secrets. As mentioned in: [Integrate a Kubernetes Cluster with an External Vault | Vault - HashiCorp Learn]

Injector created with external vault

helm install vault hashicorp/vault \
    --set "injector.externalVaultAddr=https:myvault.com"

vault secret

VAULT_HELM_SECRET_NAME=$(kubectl get secrets --output=json | jq -r '.items[].metadata | select(.name|startswith("vault-token-")).name')

Enabled vaulth auth

vault auth enable kubernetes

JWT, CA_CRT and k8s Host

TOKEN_REVIEW_JWT=$(kubectl get secret $VAULT_HELM_SECRET_NAME --output='go-template={{ .data.token }}' | base64 --decode)

KUBE_CA_CERT=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.certificate-authority-data}' | base64 --decode)

KUBE_HOST=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.server}')

Kubernetes config with SA, namespace details

vault write auth/kubernetes/config \
        token_reviewer_jwt="$TOKEN_REVIEW_JWT" \
        kubernetes_host="$KUBE_HOST" \
        kubernetes_ca_cert="$KUBE_CA_CERT"

Policy for vault secrets

vault policy write devwebapp - <<EOF
path "secret/data/devwebapp/config" {
  capabilities = ["read"]
}
EOF

Role creation for k8s auth

vault write auth/kubernetes/role/devweb-app \
        bound_service_account_names=internal-app \
        bound_service_account_namespaces=default \
        policies=devwebapp \
        ttl=24h

When deploy the application using below annotations:

  annotations:
    vault.hashicorp.com/agent-inject: 'true'
    vault.hashicorp.com/role: 'devweb-app'
    vault.hashicorp.com/agent-inject-secret-credentials.txt: 'secret/data/devwebapp/config'

But application pod is stuck at Init:0/1 and when i check the vault-agent-init logs:

2021-07-25T06:12:43.359Z [INFO]  sink.file: creating file sink
2021-07-25T06:12:43.359Z [INFO]  sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r-----
2021-07-25T06:12:43.361Z [INFO]  sink.server: starting sink server
2021-07-25T06:12:43.361Z [INFO]  template.server: starting template server
2021-07-25T06:12:43.361Z [INFO]  auth.handler: starting auth handler
[INFO] (runner) creating new runner (dry: false, once: false)
2021-07-25T06:12:43.361Z [INFO]  auth.handler: authenticating
[INFO] (runner) creating watcher
2021-07-25T06:13:43.362Z [ERROR] auth.handler: error authenticating: error="context deadline exceeded" backoff=1s
2021-07-25T06:13:44.363Z [INFO]  auth.handler: authenticating

auth.handler: error authenticating: error=“context deadline exceeded” backoff=1s
This is the error it is showing . Any help would be appreciable. spent couple of days on this :neutral_face:

I was able to fix the issue by downgrading my minikube kubernetes version from 1.21 to 1.18.17 .

Is vault pod annotation setup is not supporting latest k8s version??

If anybody facing same issue can refere github issues:
there is a jwt token structure update in k8s version 1.21:

:slightly_smiling_face: