We had IKS IBM CA cert rotation and after CA cert rotation we have updated the cert in our vault K8 auth to use the new cert and then in multiple namespaces only two namespaces having the issue to authenticate to the vault using the tokens from the pods inside these namespaces. Then we copied the token from working namespace and exported it as env and used the copied token which our vault authentication started working for us.
For example in this cluster n2 and n3 namespace all pods which try to get vault secret using the token from the pod /var/run/secrets/kubernetes.io/serviceaccount/token failed to authenticate to our vault. And in same cluster n1 namespace all pods able to authenticate to our vault using this /var/run/secrets/kubernetes.io/serviceaccount/token. So we copied this token and copied inside deployments of n2 and n3 and then the authentication started working.
Can some one help us to figure out what is causing the issue? The same config in qa environment with similar vault k8 auth config working after the cert rotation and replacement in vault.
Our vault is running inside same IKS cluster where the application pods are trying to access the secrets.
And our vault k8 auth config is something like
vault read auth/kubernetes/config
Key Value
--- -----
disable_iss_validation true
disable_local_ca_jwt true
issuer n/a
kubernetes_ca_cert <ca cert from k8s cluster>
kubernetes_host https://kubernetes.default.svc
pem_keys []
token_reviewer_jwt_set false
use_annotations_as_alias_metadata false
And the role we are using already configured for the default sa and respected namespaces n1; n2; n3
Regards
Kalesha G