ttksm
December 29, 2020, 12:39pm
1
I’ve tried to deploy Vault with UI on Amazon EKS in according with Vault on Kubernetes Deployment Guide .
When I enabled Kubernetes Auth Method, I configured parameters which Kubernetes host
is API Server Endpoint of EKS, Kubernetes CA Certificate
is CA Certificate on EKS or Vault Server Pod, and Token Reviewer JWT
is data.token
of Secret owned by ServiceAccount of Vault Server Pod.
And when I created role, I configured parameters which Bound service account names
is *, Bound service account namespaces
is *, and Generated Token's Policies
is a policy as follows.
path "secret/*" {
capabilities = ["read", "list"]
}
path "internal/data/database/config" {
capabilities = ["read"]
}
Then, I’ve deploy application like as follows.
---
apiVersion: v1
kind: Namespace
metadata:
name: sample-app
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: sample-app-sa
namespace: sample-app
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: sample-app-sa-rbac
namespace: sample-app
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: sample-app-sa
namespace: sample-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: sample-app-deployment
namespace: sample-app
spec:
replicas: 1
selector:
matchLabels:
app: sample-app
template:
metadata:
labels:
app: sample-app
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/role: "sample-app-role"
vault.hashicorp.com/agent-inject-status: "update"
vault.hashicorp.com/agent-inject-secret-credentials: "internal/data/database/config"
vault.hashicorp.com/agent-inject-template-credentials: |
{{- with secret "internal/data/database/config" -}}
USERNAME={{ .Data.data.username }}
PASSWORD={{ .Data.data.password }}
{{- end -}}
spec:
serviceAccountName: sample-app-sa
containers:
- name: nginx-container
image: nginx:1.19
Note that helm resources, vault server pods and vault-agent-injector, have been deployed on vault
namespace and kv secrets engine has been enabled.
I think I have done settings needed, however the following error occurs on vault-agent-init container on application pod.
2020-12-29T11:50:04.706Z [ERROR] auth.handler: error authenticating: error="Error making API request.
URL: PUT http://vault.vault.svc:8200/v1/auth/kubernetes/login
Code: 403. Errors:
* permission denied" backoff=1.5245839289999998
Have I configured parameters not enough?
jlj7
December 29, 2020, 1:29pm
2
Hi!
I’m in over my head, but nothing jumped out as obviously missing.
Do either of this troubleshooting suggestions apply in your case?
opened 09:22AM - 03 Jan 20 UTC
closed 10:21AM - 08 Jan 20 UTC
Hi!
I was playing around with the secret injection that was released on 19th of December. I'm not yet sure if I...
Or
opened 11:30AM - 20 Dec 19 UTC
closed 11:49AM - 27 Mar 20 UTC
The scenario we want to support is to use a vault server which pre-exists the kubernetes cluster. We want the vault-k8s...
docs
enhancement
I was also wondering about these considerations, from the second link. (Looks like this portion became the Learn tutorial on the same subject.)
Other Things Worth Noting
…
Errors for both service account and namespace will be manifest themselves as pods perpetually in the Init phase, never reaching ready. Errors are found within the vault-agent-init container logs, detailing the authentication error in clear English.
The token, both init and sidecar containers use to communicate with Vault, lives locally within the container at the following path: /home/vault/.token. Unsurprisingly, the token is not mounted into the primary container within the pod, making direct communications between Vault and primary container difficult.
ttksm
December 30, 2020, 3:30pm
3
Thank you for your reply!
First, I’m trying to deploy vault with vault-agent-injector to the single EKS cluster using official helm chart, so my case isn’t thought to match second link case such as using external vault cluster.
I wondered about your quote:
The token, both init and sidecar containers use to communicate with Vault, lives locally within the container at the following path: /home/vault/.token.
so that I investigated that path within the init container such as the following.
$ kubectl -n sample-app exec -it sample-app-deployment -c vault-agent-init -- cat /home/vault/.token
cat: can't open '/home/vault/.token': No such file or directory
Does this have any problems?
Next, I investigated internal DNS configurations of init container, discussed in the first link.
$ kubectl -n sample-app exec -it sample-app-deployment -c vault-agent-init -- cat /etc/resolv.conf
nameserver 172.20.0.10
search sample-app.svc.cluster.local svc.cluster.local cluster.local ap-northeast-1.compute.internal
I thought this indicates http://vault.vault.svc:8200
might not be resolved name in the init container.
Then, although I tried to resolve name of vault server pod as http://vault.vault.svc.cluster.local:8200
by editing injector.externalVaultAddr
in the helm chart, 403 error occured.
Hello,
Were you able to achieve authenticating to K8s? I have the same errors, same problems, Vault cannot auth to Kubernetes.
k logs -n example-app basic-secret-7d6777cdb8-zx2wt -c vault-agent-init
2022-10-30T16:10:29.741Z [INFO] sink.file: creating file sink
==> Vault agent started! Log data will stream in below:
==> Vault agent configuration:
Cgo: disabled
Log Level: info
Version: Vault v1.12.0, built 2022-10-10T18:14:33Z
Version Sha: 558abfa75702b5dab4c98e86b802fb9aef43b0eb
2022-10-30T16:10:29.741Z [INFO] sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r-----
2022-10-30T16:10:29.742Z [INFO] template.server: starting template server
2022-10-30T16:10:29.742Z [INFO] auth.handler: starting auth handler
2022-10-30T16:10:29.743Z [INFO] auth.handler: authenticating
2022-10-30T16:10:29.743Z [INFO] (runner) creating new runner (dry: false, once: false)
2022-10-30T16:10:29.743Z [INFO] sink.server: starting sink server
2022-10-30T16:10:29.745Z [WARN] (clients) disabling vault SSL verification
2022-10-30T16:10:29.745Z [WARN] (clients) disabling nomad SSL verification
2022-10-30T16:10:29.745Z [INFO] (runner) creating watcher
2022-10-30T16:10:29.833Z [ERROR] auth.handler: error authenticating:
error=
| Error making API request.
|
| URL: PUT https://vault.vault.svc:8200/v1/auth/kubernetes/login
| Code: 403. Errors:
|
| * permission denied
backoff=1s
2022-10-30T16:10:30.833Z [INFO] auth.handler: authenticating
2022-10-30T16:10:30.866Z [ERROR] auth.handler: error authenticating:
error=
| Error making API request.
|
| URL: PUT https://vault.vault.svc:8200/v1/auth/kubernetes/login
| Code: 403. Errors:
|
| * permission denied
backoff=1.72s
2022-10-30T16:10:32.592Z [INFO] auth.handler: authenticating
2022-10-30T16:10:32.603Z [ERROR] auth.handler: error authenticating:
error=
| Error making API request.
|
| URL: PUT https://vault.vault.svc:8200/v1/auth/kubernetes/login
| Code: 403. Errors:
|
| * permission denied
backoff=2.69s
2022-10-30T16:10:35.294Z [INFO] auth.handler: authenticating
2022-10-30T16:10:35.306Z [ERROR] auth.handler: error authenticating:
error=
| Error making API request.
|
| URL: PUT https://vault.vault.svc:8200/v1/auth/kubernetes/login
| Code: 403. Errors:
|
| * permission denied
backoff=4.44s
2022-10-30T16:10:39.756Z [INFO] auth.handler: authenticating
2022-10-30T16:10:39.769Z [ERROR] auth.handler: error authenticating:
error=
| Error making API request.
|
| URL: PUT https://vault.vault.svc:8200/v1/auth/kubernetes/login
| Code: 403. Errors:
|
| * permission denied
backoff=8.15s
2022-10-30T16:10:47.927Z [INFO] auth.handler: authenticating
2022-10-30T16:10:47.940Z [ERROR] auth.handler: error authenticating:
error=
| Error making API request.
|
| URL: PUT https://vault.vault.svc:8200/v1/auth/kubernetes/login
| Code: 403. Errors:
|
| * permission denied
backoff=15.15s
2022-10-30T16:11:03.093Z [INFO] auth.handler: authenticating
2022-10-30T16:11:03.107Z [ERROR] auth.handler: error authenticating:
error=
| Error making API request.
|
| URL: PUT https://vault.vault.svc:8200/v1/auth/kubernetes/login
| Code: 403. Errors:
|
| * permission denied
backoff=23.52s
2022-10-30T16:11:26.638Z [INFO] auth.handler: authenticating
2022-10-30T16:11:26.649Z [ERROR] auth.handler: error authenticating:
error=
| Error making API request.
|
| URL: PUT https://vault.vault.svc:8200/v1/auth/kubernetes/login
| Code: 403. Errors:
|
| * permission denied
backoff=45.27s
2022-10-30T16:12:11.928Z [INFO] auth.handler: authenticating
2022-10-30T16:12:11.953Z [ERROR] auth.handler: error authenticating:
error=
| Error making API request.
|
| URL: PUT https://vault.vault.svc:8200/v1/auth/kubernetes/login
| Code: 403. Errors:
|
| * permission denied
backoff=1m17.61s
2022-10-30T16:13:29.567Z [INFO] auth.handler: authenticating
2022-10-30T16:13:29.631Z [ERROR] auth.handler: error authenticating:
error=
| Error making API request.
|
| URL: PUT https://vault.vault.svc:8200/v1/auth/kubernetes/login
| Code: 403. Errors:
|
| * permission denied
backoff=2m5.05s
2022-10-30T16:15:34.682Z [INFO] auth.handler: authenticating
2022-10-30T16:15:34.737Z [ERROR] auth.handler: error authenticating:
error=
| Error making API request.
|
| URL: PUT https://vault.vault.svc:8200/v1/auth/kubernetes/login
| Code: 403. Errors:
|
| * permission denied
backoff=3m31.36s
2022-10-30T16:19:06.101Z [INFO] auth.handler: authenticating
2022-10-30T16:19:06.126Z [ERROR] auth.handler: error authenticating:
error=
| Error making API request.
|
| URL: PUT https://vault.vault.svc:8200/v1/auth/kubernetes/login
| Code: 403. Errors:
|
| * permission denied
backoff=4m14.98s
2022-10-30T16:23:21.109Z [INFO] auth.handler: authenticating
2022-10-30T16:23:21.137Z [ERROR] auth.handler: error authenticating:
error=
| Error making API request.
|
| URL: PUT https://vault.vault.svc:8200/v1/auth/kubernetes/login
| Code: 403. Errors:
|
| * permission denied
backoff=4m40.66s
2022-10-30T16:28:01.803Z [INFO] auth.handler: authenticating
2022-10-30T16:28:01.828Z [ERROR] auth.handler: error authenticating:
error=
| Error making API request.
|
| URL: PUT https://vault.vault.svc:8200/v1/auth/kubernetes/login
| Code: 403. Errors:
|
| * permission denied
backoff=3m58.41s
Hello,
May I know what was the fix? I’m seeing same errors. Any insights will be helpful.
Thanks,
Bhagya