Minikube simple-vault-client runs but with no output

todd-dsm · February 19, 2025, 6:17pm

Doing a quick refresher with Vault on minikube.

Everything works fine until the Launch a web application section. The web app (hashieducation/simple-vault-client:latest) deploys and runs without issue; steps 1-4 are good.

However, step 5 curl http://localhost:8080 executes but there’s no output; I do get a white page so at least the UI is being served even if there’s no text. Troubleshooting steps

I’ve shelled into the container
curl http://localhost:8080: no output
installed tree and nmap

# nmap localhost
...
PORT     STATE SERVICE
8080/tcp open  http-proxy (the port is open for business)

# cat $JWT_PATH: expected output
Search for secret/data/webapp/config

Test to get expected output from find: success

# find / -type d -wholename '/etc/apt'
/etc/apt

Since there is no leading slash in secret/data/webapp/config
the search begins in the current directory:

# find . -type d -wholename 'secret/data/webapp/config'

Again, no output.

root@webapp-59c4db954f-d6p4w:/app# ls -l
-rw-r--r-- 1 root root     349 Feb 15  2023 Dockerfile
-rw-r--r-- 1 root root   11357 Feb 14  2023 LICENSE
-rw-r--r-- 1 root root    1201 Feb 17  2023 README.md
-rw-r--r-- 1 root root      28 Feb 14  2023 go.mod
drwxr-xr-x 2 root root    4096 Feb 17  2023 images
-rwxr-xr-x 1 root root 7182799 Mar  2  2023 main
-rw-r--r-- 1 root root    2967 Feb 14  2023 main.go
-rw-r--r-- 1 root root    1746 Feb 14  2023 types.go
-rw-r--r-- 1 root root     967 Feb 17  2023 vault-hello-world.yaml

But, it appears there’s no secret directory anywhere

# find / -type d -name '*secret*'
/sys/module/secretmem
/usr/share/doc/git/contrib/credential/libsecret
/run/secrets
root@webapp-59c4db954f-d6p4w:/app# tree /run/secrets/
/run/secrets/
`-- kubernetes.io
    `-- serviceaccount
        |-- ca.crt -> ..data/ca.crt
        |-- namespace -> ..data/namespace
        `-- token -> ..data/token

And the one directory with secret in the name does not have data/webapp/config under it.

Does the container have a bug or am I missing something?

If it’s a bug, what is it?

in the documentation?
vault doesn’t write to the correct path?
maybe the webapp retrieves data from the wrong location?

Please advise.

claytonsilva · February 20, 2025, 3:44pm

Hi!

Can you export logs of the container too? if there a problem with boot process, is there a log information about that.

todd-dsm · April 29, 2025, 12:05am

Hey @claytonsilva, some updates since last time:

I’m now testing on EKS; the issue persists
I’ve configured the Kubernetes auth method,
And the suggested approach in Workflows
This is the webapp deployment

To answer your questions

There doesn’t seem to be any issues with the Vault itself; the boot process looks pretty clean. Log messages don’t suggest errors.
The webapp itself logs:

2025/04/28 22:19:56 Listening on port 8080
2025/04/28 22:21:26 Received Request - Port forwarding is working.
Read JWT: eyJ...IdA
Retrieved token:  
Error getting the secret from Vault, cannot convert Data to map[string[]interface{}

Looks the process is successful up-to and including Read JWT, then fails on the attempt to retrieve data from Vault.

This is what Vault logged when during an attempt to access the webapp:

% tail -8  ~/.config/node/state/k9s/screen-dumps/arn-aws-eks-us-east-1-ASDF-cluster-collab-tools-stage/collab-tools-stage/vault-vault-0-1745878900442622000.log
{"auth":{"policy_results":{"allowed":true},"token_type":"default"},"request":{"data":{"jwt":"hmac-sha256:905c3fbe01e5866d1ef6eb4725bcc15ba0f08ff96d4d730bce7e782981973f49","role":"hmac-sha256:4fe097535102227d1307e7380d99628b2f4f0263d72481c01e41fb5c8d37508f"},"headers":{"user-agent":["Go-http-client/1.1"]},"id":"6be7794b-ea0d-2b1d-7884-d773f8e9be31","mount_accessor":"auth_kubernetes_c69cef59","mount_class":"auth","mount_point":"auth/kubernetes/","mount_running_version":"v0.21.0+builtin","mount_type":"kubernetes","namespace":{"id":"root"},"operation":"update","path":"auth/kubernetes/login","remote_address":"10.101.21.213","remote_port":60886},"time":"2025-04-28T22:21:26.333269435Z","type":"request"}
{"auth":{"policy_results":{"allowed":true},"token_type":"default"},"request":{"data":{"jwt":"hmac-sha256:905c3fbe01e5866d1ef6eb4725bcc15ba0f08ff96d4d730bce7e782981973f49","role":"hmac-sha256:4fe097535102227d1307e7380d99628b2f4f0263d72481c01e41fb5c8d37508f"},"headers":{"user-agent":["Go-http-client/1.1"]},"id":"6be7794b-ea0d-2b1d-7884-d773f8e9be31","mount_accessor":"auth_kubernetes_c69cef59","mount_class":"auth","mount_point":"auth/kubernetes/","mount_running_version":"v0.21.0+builtin","mount_type":"kubernetes","namespace":{"id":"root"},"operation":"update","path":"auth/kubernetes/login","remote_address":"10.101.21.213","remote_port":60886},"response":{"data":{"error":"hmac-sha256:2f0bd98ac4cc9286a05b1e4b359331e84cca64b9de81822da0dd7114a14396ac"},"mount_accessor":"auth_kubernetes_c69cef59","mount_class":"auth","mount_point":"auth/kubernetes/","mount_running_plugin_version":"v0.21.0+builtin","mount_type":"kubernetes"},"time":"2025-04-28T22:21:26.333661116Z","type":"response"}
{"error":"permission denied","request":{"headers":{"user-agent":["Go-http-client/1.1"]},"id":"b63a802d-0a5d-f8eb-50fa-fc998ac9615c","mount_class":"secret","mount_point":"secret/","mount_running_version":"v0.21.0+builtin","mount_type":"kv","namespace":{"id":"root"},"operation":"read","path":"secret/data/webapp/config","remote_address":"10.101.21.213","remote_port":60886},"time":"2025-04-28T22:21:26.336323777Z","type":"request"}
{"error":"1 error occurred:\n\t* permission denied\n\n","request":{"headers":{"user-agent":["Go-http-client/1.1"]},"id":"b63a802d-0a5d-f8eb-50fa-fc998ac9615c","mount_class":"secret","mount_point":"secret/","mount_running_version":"v0.21.0+builtin","mount_type":"kv","namespace":{"id":"root"},"operation":"read","path":"secret/data/webapp/config","remote_address":"10.101.21.213","remote_port":60886},"response":{"data":{"error":"hmac-sha256:96d93008e3b0c936a4cec69a973d4389bc7eda188556a6c101ca151a61577446"},"mount_class":"secret","mount_point":"secret/","mount_running_plugin_version":"v0.21.0+builtin","mount_type":"kv"},"time":"2025-04-28T22:21:26.336431912Z","type":"response"}
{"auth":{"policy_results":{"allowed":true},"token_type":"default"},"request":{"data":{"jwt":"hmac-sha256:905c3fbe01e5866d1ef6eb4725bcc15ba0f08ff96d4d730bce7e782981973f49","role":"hmac-sha256:4fe097535102227d1307e7380d99628b2f4f0263d72481c01e41fb5c8d37508f"},"headers":{"user-agent":["Go-http-client/1.1"]},"id":"a40c2b21-d17b-3a56-7668-a8b5fdd35e52","mount_accessor":"auth_kubernetes_c69cef59","mount_class":"auth","mount_point":"auth/kubernetes/","mount_running_version":"v0.21.0+builtin","mount_type":"kubernetes","namespace":{"id":"root"},"operation":"update","path":"auth/kubernetes/login","remote_address":"10.101.21.213","remote_port":60886},"time":"2025-04-28T22:21:26.456738347Z","type":"request"}
{"auth":{"policy_results":{"allowed":true},"token_type":"default"},"request":{"data":{"jwt":"hmac-sha256:905c3fbe01e5866d1ef6eb4725bcc15ba0f08ff96d4d730bce7e782981973f49","role":"hmac-sha256:4fe097535102227d1307e7380d99628b2f4f0263d72481c01e41fb5c8d37508f"},"headers":{"user-agent":["Go-http-client/1.1"]},"id":"a40c2b21-d17b-3a56-7668-a8b5fdd35e52","mount_accessor":"auth_kubernetes_c69cef59","mount_class":"auth","mount_point":"auth/kubernetes/","mount_running_version":"v0.21.0+builtin","mount_type":"kubernetes","namespace":{"id":"root"},"operation":"update","path":"auth/kubernetes/login","remote_address":"10.101.21.213","remote_port":60886},"response":{"data":{"error":"hmac-sha256:2f0bd98ac4cc9286a05b1e4b359331e84cca64b9de81822da0dd7114a14396ac"},"mount_accessor":"auth_kubernetes_c69cef59","mount_class":"auth","mount_point":"auth/kubernetes/","mount_running_plugin_version":"v0.21.0+builtin","mount_type":"kubernetes"},"time":"2025-04-28T22:21:26.457237214Z","type":"response"}
{"error":"permission denied","request":{"headers":{"user-agent":["Go-http-client/1.1"]},"id":"bdb20240-62b2-dff5-c2c1-325a9c4a0dfe","mount_class":"secret","mount_point":"secret/","mount_running_version":"v0.21.0+builtin","mount_type":"kv","namespace":{"id":"root"},"operation":"read","path":"secret/data/webapp/config","remote_address":"10.101.21.213","remote_port":60886},"time":"2025-04-28T22:21:26.459855725Z","type":"request"}
{"error":"1 error occurred:\n\t* permission denied\n\n","request":{"headers":{"user-agent":["Go-http-client/1.1"]},"id":"bdb20240-62b2-dff5-c2c1-325a9c4a0dfe","mount_class":"secret","mount_point":"secret/","mount_running_version":"v0.21.0+builtin","mount_type":"kv","namespace":{"id":"root"},"operation":"read","path":"secret/data/webapp/config","remote_address":"10.101.21.213","remote_port":60886},"response":{"data":{"error":"hmac-sha256:96d93008e3b0c936a4cec69a973d4389bc7eda188556a6c101ca151a61577446"},"mount_class":"secret","mount_point":"secret/","mount_running_plugin_version":"v0.21.0+builtin","mount_type":"kv"},"time":"2025-04-28T22:21:26.46010449Z","type":"response"}

It looks like the first issue is
"error":"permission denied"... "mount_class":"secret","mount_point":"secret/","mount_running_version":"v0.21.0+builtin","mount_type":"kv","namespace":{"id":"root"},"operation":"read","path":"secret/data/webapp/config"

which would be true. The data is actually stored at:

% vault kv get secret/dct/test
==== Secret Path ====
secret/data/dct/test
...

The next error says, that it’s erroring because webapp is trying to access "path":"secret/data/webapp/.

I reran the kv put command and dropped the same stuff in "secret/data/webapp/config" ,

% vault kv get secret/webapp/config
====== Secret Path ======
secret/data/webapp/config
...

Vault still logs the same error.

I’m seeing policy_results = "allowed":true so it doesn’t appear to be policy-related.

The errors seem to be related to headers = "user-agent" which happens when I refresh the page.

Now I’m all

claytonsilva · April 29, 2025, 11:08am

It seens like missing policy for kubernetes pods to access the secret.

I’m checking this link, do you create the policy?

Remember, vault has a policy “zero trust”, so you need aways set the auth process.

If you can share the deployment manifest, i will appreciate

todd-dsm · May 2, 2025, 1:57am

@claytonsilva,

Here’s the values.yaml file and the client.

I found part of the issue, there are instructions on forming the CluserRole; found this post and tried with some modifications. This ClusterRole fixed some issues:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vault-client-auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: vault-auth
  namespace: demo
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:auth-delegator
rules:
- apiGroups:
  - authentication.k8s.io
  resources:
  - tokenreviews
  verbs:
  - create
- apiGroups:
  - authorization.k8s.io
  resources:
  - subjectaccessreviews
  verbs:
  - create
- apiGroups: [""]
  resources: ["serviceaccounts"]
  verbs: ["get", "list"]

Updating with `resources: “serviceaccounts” = “get” and “list” fixed some other stuff.

The last bit are the new logs

{"auth":{"policy_results":{"allowed":true},"token_type":"default"},"request":{"data":{"jwt":"hmac-sha256:ed254940a2f741421de4cb3533a672d9d1ac54d5256c75920634e2ebf8e5b257","role":"hmac-sha256:ac8858483085a1241cfae536b88786567aff8df1dd1ecdd18f15b79f57ad6222"},"headers":{"user-agent":["Go-http-client/1.1"]},"id":"33b2a31e-dd36-56bf-5ea7-ccb16a265bb9","mount_accessor":"auth_kubernetes_9af01620","mount_class":"auth","mount_point":"auth/kubernetes/","mount_running_version":"v0.21.0+builtin","mount_type":"kubernetes","namespace":{"id":"root"},"operation":"update","path":"auth/kubernetes/login","remote_address":"10.101.21.22","remote_port":37688},"time":"2025-05-02T01:36:04.225040279Z","type":"request"}
{"auth":{"accessor":"hmac-sha256:c26daf9e63a4c5952b3317518b4c2f3890aba4d42feb891f640d6b30d92e8ac8","client_token":"hmac-sha256:c2c12a9998fbd14b3884150dc5277d03d94ee214116810d24e01fdd52f2b21f5","display_name":"kubernetes-demo-vault-auth","entity_id":"c269e722-2b42-e72c-3dc2-9b12e0c962ab","metadata":{"role":"webapp","service_account_name":"vault-auth","service_account_namespace":"demo","service_account_secret_name":"","service_account_uid":"438ade92-13cb-462b-9a5e-317ba4baa93e"},"policies":["default","webapp-kv-ro"],"token_policies":["default","webapp-kv-ro"],"token_ttl":86400,"token_type":"service"},"request":{"data":{"jwt":"hmac-sha256:ed254940a2f741421de4cb3533a672d9d1ac54d5256c75920634e2ebf8e5b257","role":"hmac-sha256:ac8858483085a1241cfae536b88786567aff8df1dd1ecdd18f15b79f57ad6222"},"headers":{"user-agent":["Go-http-client/1.1"]},"id":"33b2a31e-dd36-56bf-5ea7-ccb16a265bb9","mount_accessor":"auth_kubernetes_9af01620","mount_class":"auth","mount_point":"auth/kubernetes/","mount_running_version":"v0.21.0+builtin","mount_type":"kubernetes","namespace":{"id":"root"},"operation":"update","path":"auth/kubernetes/login","remote_address":"10.101.21.22","remote_port":37688},"response":{"auth":{"accessor":"hmac-sha256:c26daf9e63a4c5952b3317518b4c2f3890aba4d42feb891f640d6b30d92e8ac8","client_token":"hmac-sha256:c2c12a9998fbd14b3884150dc5277d03d94ee214116810d24e01fdd52f2b21f5","display_name":"kubernetes-demo-vault-auth","entity_id":"c269e722-2b42-e72c-3dc2-9b12e0c962ab","metadata":{"role":"webapp","service_account_name":"vault-auth","service_account_namespace":"demo","service_account_secret_name":"","service_account_uid":"438ade92-13cb-462b-9a5e-317ba4baa93e"},"policies":["default","webapp-kv-ro"],"token_policies":["default","webapp-kv-ro"],"token_ttl":86400,"token_type":"service"},"mount_accessor":"auth_kubernetes_9af01620","mount_class":"auth","mount_point":"auth/kubernetes/","mount_running_plugin_version":"v0.21.0+builtin","mount_type":"kubernetes"},"time":"2025-05-02T01:36:04.261897168Z","type":"response"}
{"auth":{"accessor":"hmac-sha256:c26daf9e63a4c5952b3317518b4c2f3890aba4d42feb891f640d6b30d92e8ac8","client_token":"hmac-sha256:c2c12a9998fbd14b3884150dc5277d03d94ee214116810d24e01fdd52f2b21f5","display_name":"kubernetes-demo-vault-auth","entity_id":"c269e722-2b42-e72c-3dc2-9b12e0c962ab","metadata":{"role":"webapp","service_account_name":"vault-auth","service_account_namespace":"demo","service_account_secret_name":"","service_account_uid":"438ade92-13cb-462b-9a5e-317ba4baa93e"},"policies":["default","webapp-kv-ro"],"policy_results":{"allowed":false},"token_policies":["default","webapp-kv-ro"],"token_issue_time":"2025-05-02T01:36:04Z","token_ttl":86400,"token_type":"service"},"error":"1 error occurred:\n\t* permission denied\n\n","request":{"client_id":"c269e722-2b42-e72c-3dc2-9b12e0c962ab","client_token":"hmac-sha256:6ab65e7f1ec0d705d5260488a99639df4b8c2f5d528dbe833857a3337abc499b","client_token_accessor":"hmac-sha256:c26daf9e63a4c5952b3317518b4c2f3890aba4d42feb891f640d6b30d92e8ac8","headers":{"user-agent":["Go-http-client/1.1"]},"id":"caaaf3a8-a39b-3a17-8ea5-8ba46c6d9feb","mount_class":"secret","mount_point":"secret/","mount_running_version":"v0.21.0+builtin","mount_type":"kv","namespace":{"id":"root"},"operation":"read","path":"secret/data/webapp/config","remote_address":"10.101.21.22","remote_port":37688},"time":"2025-05-02T01:36:04.264187916Z","type":"request"}
{"auth":{"accessor":"hmac-sha256:c26daf9e63a4c5952b3317518b4c2f3890aba4d42feb891f640d6b30d92e8ac8","client_token":"hmac-sha256:c2c12a9998fbd14b3884150dc5277d03d94ee214116810d24e01fdd52f2b21f5","display_name":"kubernetes-demo-vault-auth","entity_id":"c269e722-2b42-e72c-3dc2-9b12e0c962ab","metadata":{"role":"webapp","service_account_name":"vault-auth","service_account_namespace":"demo","service_account_secret_name":"","service_account_uid":"438ade92-13cb-462b-9a5e-317ba4baa93e"},"policies":["default","webapp-kv-ro"],"policy_results":{"allowed":false},"token_policies":["default","webapp-kv-ro"],"token_issue_time":"2025-05-02T01:36:04Z","token_ttl":86400,"token_type":"service"},"error":"1 error occurred:\n\t* permission denied\n\n","request":{"client_id":"c269e722-2b42-e72c-3dc2-9b12e0c962ab","client_token":"hmac-sha256:6ab65e7f1ec0d705d5260488a99639df4b8c2f5d528dbe833857a3337abc499b","client_token_accessor":"hmac-sha256:c26daf9e63a4c5952b3317518b4c2f3890aba4d42feb891f640d6b30d92e8ac8","headers":{"user-agent":["Go-http-client/1.1"]},"id":"caaaf3a8-a39b-3a17-8ea5-8ba46c6d9feb","mount_class":"secret","mount_point":"secret/","mount_running_version":"v0.21.0+builtin","mount_type":"kv","namespace":{"id":"root"},"operation":"read","path":"secret/data/webapp/config","remote_address":"10.101.21.22","remote_port":37688},"response":{"data":{"error":"hmac-sha256:80d5a24de990e724067d4e50cc7160ffd45bb9730bba4b69072bbfd7c96a7e94"},"mount_class":"secret","mount_point":"secret/","mount_running_plugin_version":"v0.21.0+builtin","mount_type":"kv"},"time":"2025-05-02T01:36:04.264382127Z","type":"response"}
{"auth":{"policy_results":{"allowed":true},"token_type":"default"},"request":{"data":{"jwt":"hmac-sha256:ed254940a2f741421de4cb3533a672d9d1ac54d5256c75920634e2ebf8e5b257","role":"hmac-sha256:ac8858483085a1241cfae536b88786567aff8df1dd1ecdd18f15b79f57ad6222"},"headers":{"user-agent":["Go-http-client/1.1"]},"id":"a9c458fa-cf86-34b1-e917-32f8dacb9ac7","mount_accessor":"auth_kubernetes_9af01620","mount_class":"auth","mount_point":"auth/kubernetes/","mount_running_version":"v0.21.0+builtin","mount_type":"kubernetes","namespace":{"id":"root"},"operation":"update","path":"auth/kubernetes/login","remote_address":"10.101.21.22","remote_port":37688},"time":"2025-05-02T01:36:04.383352402Z","type":"request"}
{"auth":{"accessor":"hmac-sha256:99fda4dc08333bc05a29f4d7ae282ecb55c25edb15e010f76f325a75d05511b3","client_token":"hmac-sha256:0ecaff5e0817819188cdda91e9a396c91d1b150e9acdbf622437b32064196150","display_name":"kubernetes-demo-vault-auth","entity_id":"c269e722-2b42-e72c-3dc2-9b12e0c962ab","metadata":{"role":"webapp","service_account_name":"vault-auth","service_account_namespace":"demo","service_account_secret_name":"","service_account_uid":"438ade92-13cb-462b-9a5e-317ba4baa93e"},"policies":["default","webapp-kv-ro"],"token_policies":["default","webapp-kv-ro"],"token_ttl":86400,"token_type":"service"},"request":{"data":{"jwt":"hmac-sha256:ed254940a2f741421de4cb3533a672d9d1ac54d5256c75920634e2ebf8e5b257","role":"hmac-sha256:ac8858483085a1241cfae536b88786567aff8df1dd1ecdd18f15b79f57ad6222"},"headers":{"user-agent":["Go-http-client/1.1"]},"id":"a9c458fa-cf86-34b1-e917-32f8dacb9ac7","mount_accessor":"auth_kubernetes_9af01620","mount_class":"auth","mount_point":"auth/kubernetes/","mount_running_version":"v0.21.0+builtin","mount_type":"kubernetes","namespace":{"id":"root"},"operation":"update","path":"auth/kubernetes/login","remote_address":"10.101.21.22","remote_port":37688},"response":{"auth":{"accessor":"hmac-sha256:99fda4dc08333bc05a29f4d7ae282ecb55c25edb15e010f76f325a75d05511b3","client_token":"hmac-sha256:0ecaff5e0817819188cdda91e9a396c91d1b150e9acdbf622437b32064196150","display_name":"kubernetes-demo-vault-auth","entity_id":"c269e722-2b42-e72c-3dc2-9b12e0c962ab","metadata":{"role":"webapp","service_account_name":"vault-auth","service_account_namespace":"demo","service_account_secret_name":"","service_account_uid":"438ade92-13cb-462b-9a5e-317ba4baa93e"},"policies":["default","webapp-kv-ro"],"token_policies":["default","webapp-kv-ro"],"token_ttl":86400,"token_type":"service"},"mount_accessor":"auth_kubernetes_9af01620","mount_class":"auth","mount_point":"auth/kubernetes/","mount_running_plugin_version":"v0.21.0+builtin","mount_type":"kubernetes"},"time":"2025-05-02T01:36:04.429724894Z","type":"response"}
{"auth":{"accessor":"hmac-sha256:99fda4dc08333bc05a29f4d7ae282ecb55c25edb15e010f76f325a75d05511b3","client_token":"hmac-sha256:0ecaff5e0817819188cdda91e9a396c91d1b150e9acdbf622437b32064196150","display_name":"kubernetes-demo-vault-auth","entity_id":"c269e722-2b42-e72c-3dc2-9b12e0c962ab","metadata":{"role":"webapp","service_account_name":"vault-auth","service_account_namespace":"demo","service_account_secret_name":"","service_account_uid":"438ade92-13cb-462b-9a5e-317ba4baa93e"},"policies":["default","webapp-kv-ro"],"policy_results":{"allowed":false},"token_policies":["default","webapp-kv-ro"],"token_issue_time":"2025-05-02T01:36:04Z","token_ttl":86400,"token_type":"service"},"error":"1 error occurred:\n\t* permission denied\n\n","request":{"client_id":"c269e722-2b42-e72c-3dc2-9b12e0c962ab","client_token":"hmac-sha256:05908faa360b57331109251d7645190054a432576a999e4c255f4e47da8157fb","client_token_accessor":"hmac-sha256:99fda4dc08333bc05a29f4d7ae282ecb55c25edb15e010f76f325a75d05511b3","headers":{"user-agent":["Go-http-client/1.1"]},"id":"063a60e3-7d3a-5354-ec00-56c88a46a0e8","mount_class":"secret","mount_point":"secret/","mount_running_version":"v0.21.0+builtin","mount_type":"kv","namespace":{"id":"root"},"operation":"read","path":"secret/data/webapp/config","remote_address":"10.101.21.22","remote_port":37688},"time":"2025-05-02T01:36:04.432232567Z","type":"request"}
{"auth":{"accessor":"hmac-sha256:99fda4dc08333bc05a29f4d7ae282ecb55c25edb15e010f76f325a75d05511b3","client_token":"hmac-sha256:0ecaff5e0817819188cdda91e9a396c91d1b150e9acdbf622437b32064196150","display_name":"kubernetes-demo-vault-auth","entity_id":"c269e722-2b42-e72c-3dc2-9b12e0c962ab","metadata":{"role":"webapp","service_account_name":"vault-auth","service_account_namespace":"demo","service_account_secret_name":"","service_account_uid":"438ade92-13cb-462b-9a5e-317ba4baa93e"},"policies":["default","webapp-kv-ro"],"policy_results":{"allowed":false},"token_policies":["default","webapp-kv-ro"],"token_issue_time":"2025-05-02T01:36:04Z","token_ttl":86400,"token_type":"service"},"error":"1 error occurred:\n\t* permission denied\n\n","request":{"client_id":"c269e722-2b42-e72c-3dc2-9b12e0c962ab","client_token":"hmac-sha256:05908faa360b57331109251d7645190054a432576a999e4c255f4e47da8157fb","client_token_accessor":"hmac-sha256:99fda4dc08333bc05a29f4d7ae282ecb55c25edb15e010f76f325a75d05511b3","headers":{"user-agent":["Go-http-client/1.1"]},"id":"063a60e3-7d3a-5354-ec00-56c88a46a0e8","mount_class":"secret","mount_point":"secret/","mount_running_version":"v0.21.0+builtin","mount_type":"kv","namespace":{"id":"root"},"operation":"read","path":"secret/data/webapp/config","remote_address":"10.101.21.22","remote_port":37688},"response":{"data":{"error":"hmac-sha256:80d5a24de990e724067d4e50cc7160ffd45bb9730bba4b69072bbfd7c96a7e94"},"mount_class":"secret","mount_point":"secret/","mount_running_plugin_version":"v0.21.0+builtin","mount_type":"kv"},"time":"2025-05-02T01:36:04.432425763Z","type":"response"}

All 8 lines are one refresh of the page.

I noticed that service_account_secret_name is unset. But, most notably, there are a few permission denied errors in there.

Also, I’m following these instructions (steps 4 and 5) for secrets storage to secret/data/webapp/config.

todd-dsm · May 2, 2025, 5:44pm

Relative to the webapp client, there’s are a few environment variables defined.

I’ve validated that there is a token at the $JWT_PATH.

# Deploy the TEST app to the demo namespace
...
      containers:
        - name: app
          image: hashieducation/simple-vault-client:latest
          imagePullPolicy: Always
          env:
            - name: VAULT_ADDR
              value: 'http://vault.vault.svc:8200'
            - name: JWT_PATH
              value: '/var/run/secrets/kubernetes.io/serviceaccount/token'
            - name: SERVICE_PORT
              value: '8080'

Topic		Replies	Views
Undefined output Vault	0	309	September 16, 2020
Unable to connect to Vault after system comes back from hibernation Vault	0	249	August 24, 2023
Unable to output the root token and unseal keys obtained while initializing vault using kubernetes_job_v1 Kubernetes k8s , hcp-terraform , vault	0	321	August 31, 2023
Vault Secrets Enginer Version 1 non existent path response Vault	3	524	December 17, 2021
Vault go client returns an empty response on Put/Patch Vault k8s , vault	2	622	December 21, 2022

Minikube simple-vault-client runs but with no output

Related topics