Vault-agent-init authentication error, context deadline exceeded

Vault
1

I’m challenging this issue for two days. I’ve tried everything that I found but no chance. Agent init container and vault itself running on same namespace. I’ve created service account in default ns and dev ns which is vault is running in and both doesn’t work. My cluster is Rancher RKE k8s v1.20 also ACE enabled with FQDN and load balancer points to it.

I’m able to reach vault from any other container in same namespace with curl.

The agent-init container error:
2021-07-31T08:50:00.859Z [ERROR] auth.handler: error authenticating: error=“context deadline exceeded” backoff=4m58.85s

I’m able to login k8s api with followings:

cat <<EOF > /tmp/ca.crt                           
-----BEGIN CERTIFICATE-----
MIICwjCCAaqgAwIBAgIBADANBgkqhkiG9w0BAQsFADASMRAwDgYDVQQDEwdrdWJl
LWNhMB4XDTIxMDYwMTIxNTA1NFoXDTMxMDUzMDIxNTA1NFowEjEQMA4GA1UEAxMH
a3ViZS1jYTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAL+VdT+vbxfC
Est3mRUvhU7sEsvmt08N1grpH4GPVRAi72HYgwbS44UT27QUcmLgjolNQBudpWDz
2QLnNj3/Y5tYE/BBcnQor4ZbBFHJKf6/FlHlrWk2Ka0jaJqbR+LRfXdKqoXadncm
3adg3WwCxUqgEHunHaGfnp0KxNRjX0y8XzSXYfDnKZsaJ5jUC4i2P14srZ68/juD
twuV8JCrHzFjb8vII9MUgGC4MpmXvCn7T+LhU/6E+wX9CruEEltRHkej7C5EQF5H
3koMMGnCjGFMK2D+Jti5yk4GFsMlo5XLIzQQnFDS7PgB5GpY2d2iuICyr/uhkpWr
uM9cho9u5Z0CAwEAAaMjMCEwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB/wQFMAMB
Af8wDQYJKoZIhvcNAQELBQADggEBAC32Kvfn4bllLpHVW+Fxk3zx7h0TLoX6kha4
+Z9JTzDpE8rAD6XqlokcbBVkUBLbd5Qn4QrThd3lxCpQXbOCK4SFYkchBYigNHZq
zHJGeCOUlpahrYqJRjxrZLGZr6F6PuLO3g1/aVbnEY1MpWcFArfIbVFCNTBDuxYb
VgF7A9n82LMaEYkCZ2xKf0arotd5GT2jOCC+16VJ1OrtvzfhcNg4UcJwMhHxlmIG
AN6r4drBZCtAz9OSPJdPQ3SSXdt9F40G9jZrSXyzDUzWPJsl/Inrt0cnEIroYIbv
PAQx9ZAHLpTdWmxnYaMRwa6zn7eEKZIWNEkCOos3ynkgiSFVp60=
-----END CERTIFICATE-----
EOF

cat <<EOF > /tmp/sa.token                         
eyJhbGciOiJSUzI1NiIsImtpZCI6ImZTS3BVUUY2YUhEdEo0ejQ0cmNtd2V4Yi1MVXRIZ0lRV2NyTDFUM3hBcjQifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZXYiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoidmF1bHQtdG9rZW5yZXZpZXctdG9rZW4tamt3NHAiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoidmF1bHQtdG9rZW5yZXZpZXciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIzY2JlMTk5OS1iZjI0LTRiMGEtODc5ZS03MTlhYmQ4NzA1ZTEiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGV2OnZhdWx0LXRva2VucmV2aWV3In0.FyZPWvtnmZ4yLZlDDwXd1OpTz9UvqfftJwFt3sy_JYpEsmhTHVbZTNYOLL5YbJq8lmuZVMSMjaxljOODisrkBROaGtFBWkz9ZoQk934VEOsgxruQqXjzto_SOajj97o6YExwcEhY8I-y0hvRlCnvjjJM3lMT-nm253_bCFTkpN8ink36IL4f9Lx6Ws-ES-hX3Y7kKleNUM5xEt3AbQCiexaXKOreCAbO_Chcyauz37cKQgCt3HNTN5AU7M3MqdrJw-UPhRQVbizXBDlTZ10mDurT_M8Ywf-OBJ1UROXRPmY6H1IjmteZ0bMtSdN-4imvcOcMjfBzoPf2_gGuTEdkBg
EOF

cat <<EOF > /tmp/kube-auth.hcl
path "secret*" {
  capabilities = ["read"]
}
EOF

vault write auth/kubernetes/config  kubernetes_host=https://kubeapiservertest.mydomain.com:6443 kubernetes_ca_cert=@/tmp/ca.crt token_reviewer_jwt=@/tmp/sa.token

vault write auth/kubernetes/role/demo     bound_service_account_names=vault-tokenreview  bound_service_account_namespaces=default     policies=kube-auth     ttl=48h

vault policy write kube-auth /tmp/kube-auth.hcl

Test k8s connectivity:

cat <<EOF > payload.json
{
      "role": "demo",  "jwt": "eyJhbGciOiJSUzI1NiIsImtpZCI6ImZTS3BVUUY2YUhEdEo0ejQ0cmNtd2V4Yi1MVXRIZ0lRV2NyTDFUM3hBcjQifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InZhdWx0LXRva2VucmV2aWV3LXRva2VuLXhybDdrIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6InZhdWx0LXRva2VucmV2aWV3Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYTEwOTU5NGItOGRhMi00ODFjLTkyMTgtOTgxYTQ3NmUyM2VkIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmRlZmF1bHQ6dmF1bHQtdG9rZW5yZXZpZXcifQ.Y48KyJ-_BixqiJrN1dyDMvfxaE8d84ppl274oCznA_X2vKLkI7aG0XTFwa1xWy-oL_nJqDh5UXoHmO-16Y9tsIc9npkjOE404jSVCxgT3NP-8WU2YR2K2zzidf5SSA4He2qwv00C0Q1ZOSDP65My9KhdqLB4LEsQFBZ1jh6erKPB262mhFJfUkRF7jv_0GMsyWFMlm7yUPQEPALJH83EPRSWGPvveEgPwV8rhJa_wi-0Rso_eWHaTfBYd456CwQolZR7Xd5zCMYmE4-7g6TbMSLpo8xNtC3T42zurG0fa3Rh6Ah85L0X8-LRkUrC5la4zNuk91DKBWuy452Hm4lCwg"
}

curl \                        
  --request POST \
  --data @payload.json \
  https://vaulttest.mydomain.com/v1/auth/kubernetes/login
zsh: correct '@payload.json' to 'payload.json' [nyae]? n
{"request_id":"4e17571d-c494-66cc-a47b-861b0e797649","lease_id":"","renewable":false,"lease_duration":0,"data":null,"wrap_info":null,"warnings":null,"auth":{"client_token":"s.TWPLMPbdHI5u2vFyhrWnLAc9","accessor":"wkojqRdbMJ1zLYh3s0FpmEip","policies":["default","kube-auth"],"token_policies":["default","kube-auth"],"metadata":{"role":"demo","service_account_name":"vault-tokenreview","service_account_namespace":"default","service_account_secret_name":"vault-tokenreview-token-xrl7k","service_account_uid":"a109594b-8da2-481c-9218-981a476e23ed"},"lease_duration":172800,"renewable":true,"entity_id":"d0e79528-c41d-f9ec-18ba-3703dfda5e21","token_type":"service","orphan":true}}

thats how i created service account and grab ca and jwt:
kubectl apply -f -<<EOH

apiVersion: rbac.authorization.k8s.io/v1beta1

kind: ClusterRoleBinding

metadata:

name: role-tokenreview-binding

namespace: default

roleRef:

apiGroup: rbac.authorization.k8s.io

kind: ClusterRole

name: system:auth-delegator

subjects:

  • kind: ServiceAccount

name: vault-tokenreview

namespace: default

EOH

export K8S_VAULT_SA_SECRET="$(kubectl get serviceaccount vault-tokenreview -o go-template=’{{ (index .secrets 0).name }}’)"

kubectl get secret ${K8S_VAULT_SA_SECRET} -o go-template=’{{index .data “ca.crt”}}’ | base64 --decode > ca.crt

kubectl get secret ${K8S_VAULT_SA_SECRET} -o go-template=’{{ .data.token }}’ | base64 --decode > sa.token

2

Can anyone help me with this case? I’m still waiting a solution.