I’m challenging this issue for two days. I’ve tried everything that I found but no chance. Agent init container and vault itself running on same namespace. I’ve created service account in default ns and dev ns which is vault is running in and both doesn’t work. My cluster is Rancher RKE k8s v1.20 also ACE enabled with FQDN and load balancer points to it.
I’m able to reach vault from any other container in same namespace with curl.
The agent-init container error:
2021-07-31T08:50:00.859Z [ERROR] auth.handler: error authenticating: error=“context deadline exceeded” backoff=4m58.85s
I’m able to login k8s api with followings:
cat <<EOF > /tmp/ca.crt
-----BEGIN CERTIFICATE-----
MIICwjCCAaqgAwIBAgIBADANBgkqhkiG9w0BAQsFADASMRAwDgYDVQQDEwdrdWJl
LWNhMB4XDTIxMDYwMTIxNTA1NFoXDTMxMDUzMDIxNTA1NFowEjEQMA4GA1UEAxMH
a3ViZS1jYTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAL+VdT+vbxfC
Est3mRUvhU7sEsvmt08N1grpH4GPVRAi72HYgwbS44UT27QUcmLgjolNQBudpWDz
2QLnNj3/Y5tYE/BBcnQor4ZbBFHJKf6/FlHlrWk2Ka0jaJqbR+LRfXdKqoXadncm
3adg3WwCxUqgEHunHaGfnp0KxNRjX0y8XzSXYfDnKZsaJ5jUC4i2P14srZ68/juD
twuV8JCrHzFjb8vII9MUgGC4MpmXvCn7T+LhU/6E+wX9CruEEltRHkej7C5EQF5H
3koMMGnCjGFMK2D+Jti5yk4GFsMlo5XLIzQQnFDS7PgB5GpY2d2iuICyr/uhkpWr
uM9cho9u5Z0CAwEAAaMjMCEwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB/wQFMAMB
Af8wDQYJKoZIhvcNAQELBQADggEBAC32Kvfn4bllLpHVW+Fxk3zx7h0TLoX6kha4
+Z9JTzDpE8rAD6XqlokcbBVkUBLbd5Qn4QrThd3lxCpQXbOCK4SFYkchBYigNHZq
zHJGeCOUlpahrYqJRjxrZLGZr6F6PuLO3g1/aVbnEY1MpWcFArfIbVFCNTBDuxYb
VgF7A9n82LMaEYkCZ2xKf0arotd5GT2jOCC+16VJ1OrtvzfhcNg4UcJwMhHxlmIG
AN6r4drBZCtAz9OSPJdPQ3SSXdt9F40G9jZrSXyzDUzWPJsl/Inrt0cnEIroYIbv
PAQx9ZAHLpTdWmxnYaMRwa6zn7eEKZIWNEkCOos3ynkgiSFVp60=
-----END CERTIFICATE-----
EOF
cat <<EOF > /tmp/sa.token
eyJhbGciOiJSUzI1NiIsImtpZCI6ImZTS3BVUUY2YUhEdEo0ejQ0cmNtd2V4Yi1MVXRIZ0lRV2NyTDFUM3hBcjQifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZXYiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoidmF1bHQtdG9rZW5yZXZpZXctdG9rZW4tamt3NHAiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoidmF1bHQtdG9rZW5yZXZpZXciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIzY2JlMTk5OS1iZjI0LTRiMGEtODc5ZS03MTlhYmQ4NzA1ZTEiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGV2OnZhdWx0LXRva2VucmV2aWV3In0.FyZPWvtnmZ4yLZlDDwXd1OpTz9UvqfftJwFt3sy_JYpEsmhTHVbZTNYOLL5YbJq8lmuZVMSMjaxljOODisrkBROaGtFBWkz9ZoQk934VEOsgxruQqXjzto_SOajj97o6YExwcEhY8I-y0hvRlCnvjjJM3lMT-nm253_bCFTkpN8ink36IL4f9Lx6Ws-ES-hX3Y7kKleNUM5xEt3AbQCiexaXKOreCAbO_Chcyauz37cKQgCt3HNTN5AU7M3MqdrJw-UPhRQVbizXBDlTZ10mDurT_M8Ywf-OBJ1UROXRPmY6H1IjmteZ0bMtSdN-4imvcOcMjfBzoPf2_gGuTEdkBg
EOF
cat <<EOF > /tmp/kube-auth.hcl
path "secret*" {
capabilities = ["read"]
}
EOF
vault write auth/kubernetes/config kubernetes_host=https://kubeapiservertest.mydomain.com:6443 kubernetes_ca_cert=@/tmp/ca.crt token_reviewer_jwt=@/tmp/sa.token
vault write auth/kubernetes/role/demo bound_service_account_names=vault-tokenreview bound_service_account_namespaces=default policies=kube-auth ttl=48h
vault policy write kube-auth /tmp/kube-auth.hcl
Test k8s connectivity:
cat <<EOF > payload.json
{
"role": "demo", "jwt": "eyJhbGciOiJSUzI1NiIsImtpZCI6ImZTS3BVUUY2YUhEdEo0ejQ0cmNtd2V4Yi1MVXRIZ0lRV2NyTDFUM3hBcjQifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InZhdWx0LXRva2VucmV2aWV3LXRva2VuLXhybDdrIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6InZhdWx0LXRva2VucmV2aWV3Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiYTEwOTU5NGItOGRhMi00ODFjLTkyMTgtOTgxYTQ3NmUyM2VkIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmRlZmF1bHQ6dmF1bHQtdG9rZW5yZXZpZXcifQ.Y48KyJ-_BixqiJrN1dyDMvfxaE8d84ppl274oCznA_X2vKLkI7aG0XTFwa1xWy-oL_nJqDh5UXoHmO-16Y9tsIc9npkjOE404jSVCxgT3NP-8WU2YR2K2zzidf5SSA4He2qwv00C0Q1ZOSDP65My9KhdqLB4LEsQFBZ1jh6erKPB262mhFJfUkRF7jv_0GMsyWFMlm7yUPQEPALJH83EPRSWGPvveEgPwV8rhJa_wi-0Rso_eWHaTfBYd456CwQolZR7Xd5zCMYmE4-7g6TbMSLpo8xNtC3T42zurG0fa3Rh6Ah85L0X8-LRkUrC5la4zNuk91DKBWuy452Hm4lCwg"
}
curl \
--request POST \
--data @payload.json \
https://vaulttest.mydomain.com/v1/auth/kubernetes/login
zsh: correct '@payload.json' to 'payload.json' [nyae]? n
{"request_id":"4e17571d-c494-66cc-a47b-861b0e797649","lease_id":"","renewable":false,"lease_duration":0,"data":null,"wrap_info":null,"warnings":null,"auth":{"client_token":"s.TWPLMPbdHI5u2vFyhrWnLAc9","accessor":"wkojqRdbMJ1zLYh3s0FpmEip","policies":["default","kube-auth"],"token_policies":["default","kube-auth"],"metadata":{"role":"demo","service_account_name":"vault-tokenreview","service_account_namespace":"default","service_account_secret_name":"vault-tokenreview-token-xrl7k","service_account_uid":"a109594b-8da2-481c-9218-981a476e23ed"},"lease_duration":172800,"renewable":true,"entity_id":"d0e79528-c41d-f9ec-18ba-3703dfda5e21","token_type":"service","orphan":true}}
thats how i created service account and grab ca and jwt:
kubectl apply -f -<<EOH
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: role-tokenreview-binding
namespace: default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: vault-tokenreview
namespace: default
EOH
export K8S_VAULT_SA_SECRET="$(kubectl get serviceaccount vault-tokenreview -o go-template=’{{ (index .secrets 0).name }}’)"
kubectl get secret ${K8S_VAULT_SA_SECRET} -o go-template=’{{index .data “ca.crt”}}’ | base64 --decode > ca.crt
kubectl get secret ${K8S_VAULT_SA_SECRET} -o go-template=’{{ .data.token }}’ | base64 --decode > sa.token