I’m using vault in k8s cluster as external one, when I hit the vault cluster endpoint using k8s auth sometimes its providing auth token but sometimes throwing below error with permission denied:
curl --request POST --data ‘{“jwt”: "’"$SA_JWT_TOKEN"’", “role”: “vault-auth”}’ https://vault.example.com/v1/auth/kubernetes/login | jq
ERROR:
{
"errors": [
"permission denied"
]
}
When I check the logs in vault server logs:
[ERROR] auth.kubernetes.auth_kubernetes_XXXXXX: login unauthorized due to: Post "https://app.example.com/apis/authentication.k8s.io/v1/tokenreviews": dial tcp: i/o timeout
this happens every now and then but not for all requests, beacuse of this my application pods are failing.
What could be the possible reason for this behaviour??