Kubernetes 1.22/Vault 1.11.2 auth permission denied

Vault
1

Hi All

I’ve been banging my head trying to setup kubernetes auth on a new cluster and I keep running into “permission denied” errors.

I’ve got an external Vault 1.11.2 instance running and I’m trying to connect a new RKE2 k8s 1.22 cluster. I followed this tutorial to get things setup.

  • created a service account in the default namespace
  • created a ClusterRole with system:auth-delegator and added that service account
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-auth-hd
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
  - kind: ServiceAccount
    name: vault-auth-hd
    namespace: default
  • grabbed the secret token for that service account
SA_SECRET_NAME=$(kubectl get secrets --output=json | jq -r '.items[].metadata | select(.name|startswith("vault-auth-hd")).name')
SA_JWT_TOKEN=$(kubectl get secret $SA_SECRET_NAME --output 'go-template={{ .data.token }}' | base64 --decode)
  • enabled a new kubernetes auth at new path
  • grabbed CA_CERT and HOST
  • applied the configuration
vault write auth/vault-k8s-rke2hd/config token_reviewer_jwt="$SA_JWT_TOKEN" kubernetes_host="$K8S_HOST" kubernetes_ca_cert="$SA_CA_CRT" issuer="https://kubernetes.default.svc.cluster.local"
  • verified auth endpoint
Key                       Value
---                       -----
disable_iss_validation    true
disable_local_ca_jwt      false
issuer                    https://kubernetes.default.svc.cluster.local
kubernetes_ca_cert        -----BEGIN CERTIFICATE-----
cert
-----END CERTIFICATE-----
kubernetes_host           https://mycluster:6443
pem_keys                  []
  • created a test namespace and service account
kubectl create namespace vault-test
kubectl create sa vault-test-sa
  • applied the config for a new role
vault write auth/vault-k8s-rke2hd/role/vault-test-role bound_service_account_names=vault-test-sa bound_service_account_namespaces=vault-test policies=test-policy ttl=24h
  • verified the role was created
Key                                 Value
---                                 -----
alias_name_source                   serviceaccount_uid
bound_service_account_names         [vault-test-sa]
bound_service_account_namespaces    [vault-test]
policies                            [test-policy]
token_bound_cidrs                   []
token_explicit_max_ttl              0s
token_max_ttl                       0s
token_no_default_policy             false
token_num_uses                      0
token_period                        0s
token_policies                      [test-policy]
token_ttl                           24h
token_type                          default
ttl                                 24h

From what I can tell everything is setup properly but trying to manually get a token using the JWT I get permission denied

VAULT_SA_NAME=$(kubectl -n vault-test get sa vault-test-sa --output jsonpath="{.secrets[*]['name']}")
SA_JWT_TOKEN=$(kubectl -n vault-test get secret $VAULT_SA_NAME --output 'go-template={{ .data.token }}' | base64 --decode)

curl -XPOST --location --data "{\"jwt\": \"${SA_JWT_TOKEN}\", \"role\": \"vault-test-role\"}" "https://myvault:8200/v1/auth/vault-k8s-rke2hd/login"

{"errors":["permission denied"]}

decoding the JWT token it looks correct

{
  "iss": "kubernetes/serviceaccount",
  "kubernetes.io/serviceaccount/namespace": "vault-test",
  "kubernetes.io/serviceaccount/secret.name": "vault-test-sa-token-j49m8",
  "kubernetes.io/serviceaccount/service-account.name": "vault-test-sa",
  "kubernetes.io/serviceaccount/service-account.uid": "894eb0bd-9869-4e7d-8073-ba00b278ad45",
  "sub": "system:serviceaccount:vault-test:vault-test-sa"
}

Can someone point me in the right direction on how to resolve this?

thanks!

2

issue turned out to be a DNS problem on the external vault.