Permission denied 403 in post /v1/auth/kubernetes/login on vault-agent-init container

ttksm · December 29, 2020, 12:39pm

I’ve tried to deploy Vault with UI on Amazon EKS in according with Vault on Kubernetes Deployment Guide.

When I enabled Kubernetes Auth Method, I configured parameters which Kubernetes host is API Server Endpoint of EKS, Kubernetes CA Certificate is CA Certificate on EKS or Vault Server Pod, and Token Reviewer JWT is data.token of Secret owned by ServiceAccount of Vault Server Pod.

And when I created role, I configured parameters which Bound service account names is *, Bound service account namespaces is *, and Generated Token's Policies is a policy as follows.

path "secret/*" {
  capabilities = ["read", "list"]
}

path "internal/data/database/config" {
  capabilities = ["read"]
}

Then, I’ve deploy application like as follows.

---
apiVersion: v1
kind: Namespace
metadata:
  name: sample-app
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sample-app-sa
  namespace: sample-app
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: sample-app-sa-rbac
  namespace: sample-app
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: sample-app-sa
  namespace: sample-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app-deployment
  namespace: sample-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "sample-app-role"
        vault.hashicorp.com/agent-inject-status: "update"
        vault.hashicorp.com/agent-inject-secret-credentials: "internal/data/database/config"
        vault.hashicorp.com/agent-inject-template-credentials: |
          {{- with secret "internal/data/database/config" -}}
          USERNAME={{ .Data.data.username }}
          PASSWORD={{ .Data.data.password }}
          {{- end -}}
    spec:
      serviceAccountName: sample-app-sa
      containers:
      - name: nginx-container
        image: nginx:1.19

Note that helm resources, vault server pods and vault-agent-injector, have been deployed on vault namespace and kv secrets engine has been enabled.

I think I have done settings needed, however the following error occurs on vault-agent-init container on application pod.

2020-12-29T11:50:04.706Z [ERROR] auth.handler: error authenticating: error="Error making API request.

URL: PUT http://vault.vault.svc:8200/v1/auth/kubernetes/login
Code: 403. Errors:

* permission denied" backoff=1.5245839289999998

Have I configured parameters not enough?

jlj7 · December 29, 2020, 1:29pm

Hi!

I’m in over my head, but nothing jumped out as obviously missing.

Do either of this troubleshooting suggestions apply in your case?

Or

I was also wondering about these considerations, from the second link. (Looks like this portion became the Learn tutorial on the same subject.)

Other Things Worth Noting

…

Errors for both service account and namespace will be manifest themselves as pods perpetually in the Init phase, never reaching ready. Errors are found within the vault-agent-init container logs, detailing the authentication error in clear English.

The token, both init and sidecar containers use to communicate with Vault, lives locally within the container at the following path: /home/vault/.token. Unsurprisingly, the token is not mounted into the primary container within the pod, making direct communications between Vault and primary container difficult.

ttksm · December 30, 2020, 3:30pm

Thank you for your reply!

First, I’m trying to deploy vault with vault-agent-injector to the single EKS cluster using official helm chart, so my case isn’t thought to match second link case such as using external vault cluster.
I wondered about your quote:

The token, both init and sidecar containers use to communicate with Vault, lives locally within the container at the following path: /home/vault/.token.

so that I investigated that path within the init container such as the following.

$ kubectl -n sample-app exec -it sample-app-deployment -c vault-agent-init -- cat /home/vault/.token
cat: can't open '/home/vault/.token': No such file or directory

Does this have any problems?

Next, I investigated internal DNS configurations of init container, discussed in the first link.

$ kubectl -n sample-app exec -it sample-app-deployment -c vault-agent-init -- cat /etc/resolv.conf
nameserver 172.20.0.10
search sample-app.svc.cluster.local svc.cluster.local cluster.local ap-northeast-1.compute.internal

I thought this indicates http://vault.vault.svc:8200 might not be resolved name in the init container.
Then, although I tried to resolve name of vault server pod as http://vault.vault.svc.cluster.local:8200 by editing injector.externalVaultAddr in the helm chart, 403 error occured.

tirelibirefe · October 30, 2022, 4:33pm

Hello,
Were you able to achieve authenticating to K8s? I have the same errors, same problems, Vault cannot auth to Kubernetes.

k logs -n example-app basic-secret-7d6777cdb8-zx2wt  -c vault-agent-init
2022-10-30T16:10:29.741Z [INFO]  sink.file: creating file sink
==> Vault agent started! Log data will stream in below:

==> Vault agent configuration:

                     Cgo: disabled
               Log Level: info
                 Version: Vault v1.12.0, built 2022-10-10T18:14:33Z
             Version Sha: 558abfa75702b5dab4c98e86b802fb9aef43b0eb

2022-10-30T16:10:29.741Z [INFO]  sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r-----
2022-10-30T16:10:29.742Z [INFO]  template.server: starting template server
2022-10-30T16:10:29.742Z [INFO]  auth.handler: starting auth handler
2022-10-30T16:10:29.743Z [INFO]  auth.handler: authenticating
2022-10-30T16:10:29.743Z [INFO] (runner) creating new runner (dry: false, once: false)
2022-10-30T16:10:29.743Z [INFO]  sink.server: starting sink server
2022-10-30T16:10:29.745Z [WARN] (clients) disabling vault SSL verification
2022-10-30T16:10:29.745Z [WARN] (clients) disabling nomad SSL verification
2022-10-30T16:10:29.745Z [INFO] (runner) creating watcher
2022-10-30T16:10:29.833Z [ERROR] auth.handler: error authenticating:
  error=
  | Error making API request.
  |
  | URL: PUT https://vault.vault.svc:8200/v1/auth/kubernetes/login
  | Code: 403. Errors:
  |
  | * permission denied
   backoff=1s
2022-10-30T16:10:30.833Z [INFO]  auth.handler: authenticating
2022-10-30T16:10:30.866Z [ERROR] auth.handler: error authenticating:
  error=
  | Error making API request.
  |
  | URL: PUT https://vault.vault.svc:8200/v1/auth/kubernetes/login
  | Code: 403. Errors:
  |
  | * permission denied
   backoff=1.72s
2022-10-30T16:10:32.592Z [INFO]  auth.handler: authenticating
2022-10-30T16:10:32.603Z [ERROR] auth.handler: error authenticating:
  error=
  | Error making API request.
  |
  | URL: PUT https://vault.vault.svc:8200/v1/auth/kubernetes/login
  | Code: 403. Errors:
  |
  | * permission denied
   backoff=2.69s
2022-10-30T16:10:35.294Z [INFO]  auth.handler: authenticating
2022-10-30T16:10:35.306Z [ERROR] auth.handler: error authenticating:
  error=
  | Error making API request.
  |
  | URL: PUT https://vault.vault.svc:8200/v1/auth/kubernetes/login
  | Code: 403. Errors:
  |
  | * permission denied
   backoff=4.44s
2022-10-30T16:10:39.756Z [INFO]  auth.handler: authenticating
2022-10-30T16:10:39.769Z [ERROR] auth.handler: error authenticating:
  error=
  | Error making API request.
  |
  | URL: PUT https://vault.vault.svc:8200/v1/auth/kubernetes/login
  | Code: 403. Errors:
  |
  | * permission denied
   backoff=8.15s
2022-10-30T16:10:47.927Z [INFO]  auth.handler: authenticating
2022-10-30T16:10:47.940Z [ERROR] auth.handler: error authenticating:
  error=
  | Error making API request.
  |
  | URL: PUT https://vault.vault.svc:8200/v1/auth/kubernetes/login
  | Code: 403. Errors:
  |
  | * permission denied
   backoff=15.15s
2022-10-30T16:11:03.093Z [INFO]  auth.handler: authenticating
2022-10-30T16:11:03.107Z [ERROR] auth.handler: error authenticating:
  error=
  | Error making API request.
  |
  | URL: PUT https://vault.vault.svc:8200/v1/auth/kubernetes/login
  | Code: 403. Errors:
  |
  | * permission denied
   backoff=23.52s
2022-10-30T16:11:26.638Z [INFO]  auth.handler: authenticating
2022-10-30T16:11:26.649Z [ERROR] auth.handler: error authenticating:
  error=
  | Error making API request.
  |
  | URL: PUT https://vault.vault.svc:8200/v1/auth/kubernetes/login
  | Code: 403. Errors:
  |
  | * permission denied
   backoff=45.27s
2022-10-30T16:12:11.928Z [INFO]  auth.handler: authenticating
2022-10-30T16:12:11.953Z [ERROR] auth.handler: error authenticating:
  error=
  | Error making API request.
  |
  | URL: PUT https://vault.vault.svc:8200/v1/auth/kubernetes/login
  | Code: 403. Errors:
  |
  | * permission denied
   backoff=1m17.61s
2022-10-30T16:13:29.567Z [INFO]  auth.handler: authenticating
2022-10-30T16:13:29.631Z [ERROR] auth.handler: error authenticating:
  error=
  | Error making API request.
  |
  | URL: PUT https://vault.vault.svc:8200/v1/auth/kubernetes/login
  | Code: 403. Errors:
  |
  | * permission denied
   backoff=2m5.05s
2022-10-30T16:15:34.682Z [INFO]  auth.handler: authenticating
2022-10-30T16:15:34.737Z [ERROR] auth.handler: error authenticating:
  error=
  | Error making API request.
  |
  | URL: PUT https://vault.vault.svc:8200/v1/auth/kubernetes/login
  | Code: 403. Errors:
  |
  | * permission denied
   backoff=3m31.36s
2022-10-30T16:19:06.101Z [INFO]  auth.handler: authenticating
2022-10-30T16:19:06.126Z [ERROR] auth.handler: error authenticating:
  error=
  | Error making API request.
  |
  | URL: PUT https://vault.vault.svc:8200/v1/auth/kubernetes/login
  | Code: 403. Errors:
  |
  | * permission denied
   backoff=4m14.98s
2022-10-30T16:23:21.109Z [INFO]  auth.handler: authenticating
2022-10-30T16:23:21.137Z [ERROR] auth.handler: error authenticating:
  error=
  | Error making API request.
  |
  | URL: PUT https://vault.vault.svc:8200/v1/auth/kubernetes/login
  | Code: 403. Errors:
  |
  | * permission denied
   backoff=4m40.66s
2022-10-30T16:28:01.803Z [INFO]  auth.handler: authenticating
2022-10-30T16:28:01.828Z [ERROR] auth.handler: error authenticating:
  error=
  | Error making API request.
  |
  | URL: PUT https://vault.vault.svc:8200/v1/auth/kubernetes/login
  | Code: 403. Errors:
  |
  | * permission denied
   backoff=3m58.41s

bhagya.bhagu · January 30, 2023, 8:05am

Hello,

May I know what was the fix? I’m seeing same errors. Any insights will be helpful.

Thanks,
Bhagya