Permission denied when try retrieve secrets from vault to k8s pod using vault-webhook

Vault
1

Hello, my vault instance in google cloud VM (not in k8s cluster) and i add k8s external API endpoint to security firewall of vault

export VAULT_ADDR='https://kind-pug-vault.xxx.com'

vault login xxx.xxx

vault secrets enable -path=secret kv-v2

vault kv put secret/k8s/test username="k8s" password="k8s_supersecretpass"

vault auth enable kubernetes

kubectl apply -f resources/vault-webhook.ServiceAccount.yaml


apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-webhook
  namespace: argocd

kubectl apply -f resources/vault-webhook.Secret.yaml


apiVersion: v1
kind: Secret
metadata:
  name: vault-webhook-token
  namespace: argocd
  annotations:
    kubernetes.io/service-account.name: vault-webhook
type: kubernetes.io/service-account-token

kubectl apply -f resources/vault-webhook/vault-webhook.ClusterRoleBinding.yaml


apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
  namespace: argocd
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
  - kind: ServiceAccount
    name: vault-webhook
    namespace: argocd



KUBE_API_SERVER=$(gcloud container clusters describe kind-pug --zone us-central1 --format "get(endpoint)")
SA_JWT_TOKEN=$(kubectl get secret vault-webhook-token -n argocd -o jsonpath="{.data.token}" | base64 --decode)
SA_CA_CRT=$(kubectl get secret vault-webhook-token -n argocd -o jsonpath="{.data['ca\.crt']}" | base64 --decode)



vault write auth/kubernetes/config \
   token_reviewer_jwt="$SA_JWT_TOKEN" \
   kubernetes_host="https://$KUBE_API_SERVER" \
   kubernetes_ca_cert="$SA_CA_CRT"

vault policy write k8s-read-policy - <<EOF
path "*" {
 capabilities = ["read", "list"]
}
EOF

vault write auth/kubernetes/role/k8s-read-role \
   bound_service_account_names=vault-webhook \
   bound_service_account_namespaces=argocd \
   policies=k8s-read-policy \
   ttl=24h

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
 name: vault-var
 namespace: argocd
 annotations:
   vault.security.banzaicloud.io/vault-addr: "https://kind-pug-vault.xxx.com"
   vault.security.banzaicloud.io/vault-role: "k8s-read-role"
   vault.security.banzaicloud.io/vault-skip-verify: "true"
spec:
 serviceAccountName: vault-webhook
 containers:
 - name: vault-var
   image: busybox
   command: ["sh", "-c", "echo $MYSECRET && echo going to sleep... && sleep 10000"]
   env:
   - name: MYSECRET
     value: vault:secret/data/k8s/test#password
EOF

So error from pod is

time="2024-07-15T14:33:50Z" level=error msg="failed to request new Vault token" app=vault-env err="Error making API request.\n\nURL: PUT https://kind-pug-vault.xxx.com/v1/auth/kubernetes/login\nCode: 403. Raw Message:\n\n<!doctype html><meta charset=\"utf-8\"><meta name=viewport content=\"width=device-width, initial-scale=1\"><title>403</title>403 Forbidden"

but if i do curl


curl --request POST --data '{"jwt": "eyJhbGciOiJSUzI1NiIsImtpZCI6ImpUY0lyREVtU1cyNDNfX3lqOUdzdlY5YWtjTVdUY2JReHdDR1RFMHRpczAifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50xxxxiZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJhcmdvY2QiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdxxxxL3NlY3JldCxxxlIjoidmF1bHQtd2ViaG9vay10b2tlbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdxxxxbWUiOiJ2YXVsdC13ZWJob29rIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnxxxxkIjoiYTUxMjgzZGYtOTY5ZS00OxxxOTAtNzRmNzMyMGVmZDMxIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmFyZ29jZDp2YXVsdC13ZWJob29rInxxxxg1lQdWS6I-wJRgTEThHsLR-W4C4NYGzCDFcuv6zw1XJ7PaFNJsA4jc5d3_driwISwwmV1OsUyyK9ju1vBRAaDQ8UJREusmuvmj3OxYt7de0FO_Dxxx7IXxJXiPwN6KKMbo6Xxxx6vOxEH-Tgk9IDWJSquYmLeHt6-krmTVFlehFAVZlPJZTXoxxxxMO7ZbU4m4HmNXL1R2xxxxoBki5qcp984qxVmc7ml8nKWvhcmuf7pID3tfixaJhxxxUCi2P4ej3F7TSCwEKkJf2fefjRwxluTFKzaTVXfDmp955NeGr_lIk8cOAZGQaWf0NGHQGF8NhuAl6l4uA", "role": "k8s-read-role"}' https://kind-pug-vault.xxx.com/v1/auth/kubernetes/login

i receive

{"request_id":"bca73211-ae5f-b22f-7999-9b0ea71a3445","lease_id":"","renewable":false,"lease_duration":0,"data":null,"wrap_info":null,"warnings":["TTL of \"24h\" exceeded the effective max_ttl of \"30m\"; TTL value is capped accordingly"],"auth":{"client_token":"hvs.CAESIB3pxxYoXCVvUxax-VP8SB95axoGsnQecnxx0F2lIDca0Gh4KHGh2cy5yaUxx3M4QWd4Z1JmxxCT1VjRGRoczU","accessor":"r4JYGTqL7xxxqN507RjWxyi0","policies":["default","k8s-read-policy"],"token_policies":["default","k8s-read-policy"],"metadata":{"role":"k8s-read-role","service_account_name":"vault-webhook","service_account_namespace":"argocd","service_account_secret_name":"vault-webhook-token","service_account_uid":"a51283df-9xxe-493e-bx90-74f7320xxfd31"},"lease_duration":1800,"renewable":true,"entity_id":"6cxefad4-7xb7-0e1x-21x2-6407913xb135","token_type":"service","orphan":true,"mfa_requirement":null,"num_uses":0}}
2

Problem was in GCP Load balancer which used for balancing traffice to vault group instances

so i add ip address from k8s cloud nat to allow in load balancer

So it was network problem