Permission denied 403 in post /v1/auth/kubernetes/login on vault-agent-init container

I’ve tried to deploy Vault with UI on Amazon EKS in according with Vault on Kubernetes Deployment Guide.

When I enabled Kubernetes Auth Method, I configured parameters which Kubernetes host is API Server Endpoint of EKS, Kubernetes CA Certificate is CA Certificate on EKS or Vault Server Pod, and Token Reviewer JWT is data.token of Secret owned by ServiceAccount of Vault Server Pod.

And when I created role, I configured parameters which Bound service account names is *, Bound service account namespaces is *, and Generated Token's Policies is a policy as follows.

path "secret/*" {
  capabilities = ["read", "list"]
}

path "internal/data/database/config" {
  capabilities = ["read"]
}

Then, I’ve deploy application like as follows.

---
apiVersion: v1
kind: Namespace
metadata:
  name: sample-app
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sample-app-sa
  namespace: sample-app
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: sample-app-sa-rbac
  namespace: sample-app
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: sample-app-sa
  namespace: sample-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app-deployment
  namespace: sample-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "sample-app-role"
        vault.hashicorp.com/agent-inject-status: "update"
        vault.hashicorp.com/agent-inject-secret-credentials: "internal/data/database/config"
        vault.hashicorp.com/agent-inject-template-credentials: |
          {{- with secret "internal/data/database/config" -}}
          USERNAME={{ .Data.data.username }}
          PASSWORD={{ .Data.data.password }}
          {{- end -}}
    spec:
      serviceAccountName: sample-app-sa
      containers:
      - name: nginx-container
        image: nginx:1.19

Note that helm resources, vault server pods and vault-agent-injector, have been deployed on vault namespace and kv secrets engine has been enabled.

I think I have done settings needed, however the following error occurs on vault-agent-init container on application pod.

2020-12-29T11:50:04.706Z [ERROR] auth.handler: error authenticating: error="Error making API request.

URL: PUT http://vault.vault.svc:8200/v1/auth/kubernetes/login
Code: 403. Errors:

* permission denied" backoff=1.5245839289999998

Have I configured parameters not enough?

Hi!

I’m in over my head, but nothing jumped out as obviously missing.

Do either of this troubleshooting suggestions apply in your case?

Or

I was also wondering about these considerations, from the second link. (Looks like this portion became the Learn tutorial on the same subject.)

Other Things Worth Noting

Errors for both service account and namespace will be manifest themselves as pods perpetually in the Init phase, never reaching ready. Errors are found within the vault-agent-init container logs, detailing the authentication error in clear English.

The token, both init and sidecar containers use to communicate with Vault, lives locally within the container at the following path: /home/vault/.token. Unsurprisingly, the token is not mounted into the primary container within the pod, making direct communications between Vault and primary container difficult.

Thank you for your reply!

First, I’m trying to deploy vault with vault-agent-injector to the single EKS cluster using official helm chart, so my case isn’t thought to match second link case such as using external vault cluster.
I wondered about your quote:

The token, both init and sidecar containers use to communicate with Vault, lives locally within the container at the following path: /home/vault/.token.

so that I investigated that path within the init container such as the following.

$ kubectl -n sample-app exec -it sample-app-deployment -c vault-agent-init -- cat /home/vault/.token
cat: can't open '/home/vault/.token': No such file or directory

Does this have any problems?

Next, I investigated internal DNS configurations of init container, discussed in the first link.

$ kubectl -n sample-app exec -it sample-app-deployment -c vault-agent-init -- cat /etc/resolv.conf
nameserver 172.20.0.10
search sample-app.svc.cluster.local svc.cluster.local cluster.local ap-northeast-1.compute.internal

I thought this indicates http://vault.vault.svc:8200 might not be resolved name in the init container.
Then, although I tried to resolve name of vault server pod as http://vault.vault.svc.cluster.local:8200 by editing injector.externalVaultAddr in the helm chart, 403 error occured.