Vault server running in a K8S cluster is unable to talk to another K8S cluster for verifying the token ( Token Review API fails)

Vault
1

Hi,
I have a set up where vault is running on a k8s cluster and the vault agent injector is running on another cluster ( Application cluster) .

Below are the configurations

On the Application Cluster

##Install the vault agent injector

helm install -n vault --create-namespace -g vault/ --set "injector.externalVaultAddr=http://external-vault.vault:31258"

external-vault is an endpoint pointing to the address of the vault server and this connectivity is working fine

Create a service account and cluster role binding for the token review account

apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-auth
  namespace: vault
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
  namespace: vault
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: vault-auth
  namespace: vault

Export the information to configure the vault k8s auth

export SA_SECRET_NAME=vault-auth-token-****

export SA_JWT_TOKEN=$(kubectl get secret $SA_SECRET_NAME \
    -n vault --output 'go-template={{ .data.token }}' | base64 --decode)

export SA_CA_CRT=$(kubectl config view --raw --minify --flatten \
    -n vault --output 'jsonpath={.clusters[].cluster.certificate-authority-data}' | base64 --decode)

export K8S_HOST=$(kubectl config view --raw --minify --flatten \
    --output 'jsonpath={.clusters[].cluster.server}')

On the vault server cluster

Enable K8s auth

vault auth enable kubernetes

Configure auth

vault write auth/kubernetes/config \
     token_reviewer_jwt="$SA_JWT_TOKEN" \
     kubernetes_host="$K8S_HOST" \
     kubernetes_ca_cert="$SA_CA_CRT" \
    disable_local_ca_jwt="true" \
   disable_iss_validation="true"

When i examine the logs of the vault server with the debug mode enabled , i see this error

2022-12-06T09:01:30.368Z [DEBUG] auth.kubernetes.auth_kubernetes_e43f34af: login unauthorized: err=“Post "https://192.168.20.30:443/apis/authentication.k8s.io/v1/tokenreviews\”: dial tcp ********:443: i/o timeout"

However i am able to curl and get the output from the VM in the same network

Input

curl -k -X "POST" "https://192.168.20.30:443/apis/authentication.k8s.io/v1/tokenreviews" \
     -H 'Authorization: Bearer ...' \
     -H 'Content-Type: application/json; charset=utf-8' \
     -d $'{
  "kind": "TokenReview",
  "apiVersion": "authentication.k8s.io/v1",
  "spec": {
    "token": "..."  
    }
}'

Output

{
  "kind": "TokenReview",
  "apiVersion": "authentication.k8s.io/v1",
  "metadata": {
    "creationTimestamp": null,
    "managedFields": [
      {
        "manager": "curl",
        "operation": "Update",
        "apiVersion": "authentication.k8s.io/v1",
        "time": "2022-12-06T11:32:17Z",
        "fieldsType": "FieldsV1",
        "fieldsV1": {
          "f:spec": {
            "f:token": {}
          }
        }
      }
    ]
  },
  "spec": {
    "token": "..."
  },
  "status": {
    "authenticated": true,
    "user": {
      "username": "system:serviceaccount:zbi-platform:internal-app",
      "uid": "43871d65-3835-4b4d-b3fe-eb6e93a28ca5",
      "groups": [
        "system:serviceaccounts",
        "system:serviceaccounts:zbi-platform",
        "system:authenticated"
      ]
    },
    "audiences": [
      "https://kubernetes.default.svc.cluster.local"
    ]
  }
}

How do I go about debugging the issue ?