Retrieving secret from Enterprise vault (with custom namespace) fails with :403 Forbidden: "{"errors":["permission denied”]}”

Deployed and configured vault using below steps:

1# Deployed helm using Enterprise trial license in k8 cluster(namespace: k8ns)
kubectl create secret generic vault-ent-license --from-literal=“license=${secret}”
helm install hashicorp hashicorp/vault -f config.yaml -n k8ns

2# Initialized vault and unsealed keys
kubectl exec hashicorp-vault-0 -n k8ns – vault operator init -format=json
kubectl exec hashicorp-vault-0 -n k8ns – vault operator unseal key1
kubectl exec hashicorp-vault-0 -n k8ns – vault operator unseal key2
kubectl exec hashicorp-vault-0 -n k8ns – vault operator unseal key3

3# Performed vault login
kubectl exec hashicorp-vault-0 -n k8ns – vault login

4# Created vault namespace
created vault namespace(vaultns) from vault UI

5# Created secret engine kv and created ACL policy
kubectl exec hashicorp-vault-0 -n k8ns – vault secrets enable -namespace=“vaultns” -version=2 -path=kv kv

Added policy(k8s_test) with below capabilities
path “kv/*”
{
capabilities = [“create”, “read”, “update”, “delete”, “list”, “sudo”]
}

6# Kubernetes authentication method to use the service account token
kubectl exec hashicorp-vault-0 -n k8ns – vault auth enable -namespace=vaultns -path=“kubernetes” kubernetes
kubectl exec hashicorp-vault-0 -n k8ns – vault write -namespace=vaultns auth/kubernetes/config token_reviewer_jwt=“” kubernetes_host=https://kubernetes.k8ns.svc kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt issuer=“https://kubernetes.k8ns.svc.cluster.local

7# Created roles and added secrets
kubectl exec hashicorp-vault-0 – vault write -namespace=vaultns auth/kubernetes/role/product bound_service_account_names=hashicorp-vault bound_service_account_namespaces=k8ns policies=k8s_test ttl=1h
kubectl exec hashicorp-vault-0 – vault kv put -namespace=vaultns secret/a/product/system/database username=system password=R@spb3rry123

8# Configure vault in application
vault:
url: http://hashicorp-vault.k8ns.svc:8200
namespace: vaultns
authenticationPath: kubernetes
secretEngine: kv

9# Below is the Request and headers and error returned:
vaultUri:http://hashicorp-vault.k8ns.svc:8200, serviceName:product
“GET /v1/kv/data/a/product/system/database HTTP/1.1[\r][\n]”
“Accept: application/json, application/*+json[\r][\n]”
“X-Vault-Namespace: vaultns[\r][\n]”
“Accept-Encoding: gzip, x-gzip, deflate[\r][\n]”
“Host: hashicorp-vault.k8ns.svc:8200[\r][\n]”
“Connection: keep-alive[\r][\n]”
“User-Agent: Apache-HttpClient/5.2.3 (Java/17.0.10)[\r][\n]”
“[\r][\n]”
“HTTP/1.1 403 Forbidden[\r][\n]”
“Cache-Control: no-store[\r][\n]”
“Content-Type: application/json[\r][\n]”
“Strict-Transport-Security: max-age=31536000; includeSubDomains[\r][\n]”
“X-Vault-Namespace: vaultns[\r][\n]”
“Date: Thu, 09 May 2024 05:22:55 GMT[\r][\n]”
“Content-Length: 33[\r][\n]”
“[\r][\n]”
“{“errors”:[“permission denied”]}[\n]”
Exception!!! org.springframework.web.client.HttpClientErrorException$Forbidden: 403 Forbidden: “{“errors”:[“permission denied”]}” Status 403 Forbidden [kv]: permission denied

Try to enable audit logging in Vault and check messages on auth request

Below is the vault log I found-

{"auth":{"token_type":"default"},"error":"permission denied","request":{"id":"368b8a70-0639-c147-77bd-42f113ee451c","mount_class":"secret","mount_point":"vaultns/kv/","mount_type":"kv","mount_running_version":"v0.17.0+builtin","namespace":{"id":"jA0BO","path":"vaultns/"},"operation":"read","path":"kv/data/a/product/system/database","remote_address":"10.1.26.25","remote_port":44994,"request_uri":"/v1/kv/data/a/product/system/database"},"time":"2024-05-16T08:49:04.3743163Z","type":"request"}
{"auth":{"token_type":"default"},"error":"1 error occurred:\n\t* permission denied\n\n","time":"2024-05-16T08:49:04.3748239Z","type":"response","request":{"id":"368b8a70-0639-c147-77bd-42f113ee451c","mount_class":"secret","mount_point":"vaultns/kv/","mount_type":"kv","mount_running_version":"v0.17.0+builtin","namespace":{"id":"jA0BO","path":"vaultns/"},"operation":"read","path":"kv/data/a/product/system/database","remote_address":"10.1.26.25","remote_port":44994,"request_uri":"/v1/kv/data/a/product/system/database"},"response":{"data":{"error":"hmac-sha256:f2cfa789607d33d14e5763ac4657e05960b8db3a4b341969d04b306d5e5d45de"},"mount_class":"secret","mount_point":"vaultns/kv/","mount_running_plugin_version":"v0.17.0+builtin","mount_type":"kv"}}

Your policy points to kv mount point, but you requesting vaultns/kv with durgemprvault namespace


"request": {
        "id": "368b8a70-0639-c147-77bd-42f113ee451c",
        "mount_class": "secret",
        "mount_point": "vaultns/kv/",
        "mount_type": "kv",
        "mount_running_version": "v0.17.0+builtin",
        "namespace": {
            "id": "jA0BO",
            "path": "durgemprvault/"
        },

sorry for the typo: Intention was to hide usernames from the post. I have updated the request/response

Anyway, I believe you don’t need the namespace pointing to the path:

"mount_point": "vaultns/kv/"

only kv as a mount point.
You already have namespace as an attribute.

Thank you for your responses.
I am not sure of how that mount_point is being set.
Here is my request response and audit log.

curl --location --request GET 'http://localhost:8200/v1/kv/data/product/a/system/database' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--header 'X-Vault-Namespace: vaultns' \
--header 'Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6Ilpsb2N4U3VJV3R2UDdGcjcxYjIyc0xncXpPZmZzT3F5Smh1SUlDMVlxajgifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkdXJnZW1wciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJ2YXVsdC1hdXRoIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6InZhdWx0LWF1dGgiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI2NDg3YmI2NS0wYzFhLTRkZTUtOWFjZS1jZmZiNjhmNzZjZDQiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZHVyZ2VtcHI6dmF1bHQtYXV0aCJ9.PmJwJ_UOuyg6NMoPUhSAUDM1uJaLrJiT1zvPnic5Uhfw0dH1xr6nnbf3AJYTEtvTNZpvKNIcGh0f_HIb_NPl-Guzos2CkoGb2fq_5oh22prPBcEiKZ2PI0rQCOkFErO5Y5DEDK7DyztGQKkVL5gn19JGin6akISK4risD91f4I0obi9OzcX3N6e2epzXzXBsLIhi8JnOmZmcLUnzwyICe1C0MVPo4L0BRy_gKL0Hm0yMUQuYEbgA56MOs95FtlaFuIC9VxTGfUDNcaNnUcGFV70HhO79cmQN26B2MkWjPiE0BMuqhV8ZaufxoYltVXwYzrtpX_tp9FzelBiyIXZS2g' \

response:
{"errors":["permission denied"]}

audit-log:
{"auth":{"token_type":"default"},"error":"1 error occurred:\n\t* permission denied\n\n","time":"2024-05-20T11:01:18.2782614Z","type":"response","request":{"id":"71abc704-2163-9a55-b9ce-bf934c59670c","mount_class":"secret","mount_point":"vaultns/kv/","mount_type":"kv","mount_running_version":"v0.17.0+builtin","namespace":{"id":"e2mJB","path":"vaultns/"},"operation":"read","path":"kv/data/product/a/system/database","remote_address":"10.1.26.237","remote_port":35912,"request_uri":"/v1/kv/data/product/a/system/database"},"response":{"data":{"error":"hmac-sha256:c853ef7b74f1fafea456105412b814cec427885132bd17806d4baeb8dc2ef0a4"},"mount_class":"secret","mount_point":"vaultns/kv/","mount_running_plugin_version":"v0.17.0+builtin","mount_type":"kv"}}

Maybe the path with the namespace need to be fixed, check here: https://discuss.hashicorp.com/t/how-to-use-namespaces-in-kv-v2/24007

Also, are you sure, that this is really an access token?
Seems more like a JWT token. For JWT auth you need request an access token first (starts with hvs.XXX), then use it to read secrets.
curl -s --header "X-Vault-Token: hvs.CAy5ZsZEM....piR25vW" .......

Here, our requirement is to retrieve secret from Ent vault namespace using k8 authentication which will use JWT token. Our intension is not to use root token(hvc.abc…).

Sure, you don’t need (and should not) to use a root token.
Here I’m referring to a normal token, issued by Vault as you authenticate (with JWT, k8s etc. methods)

Can someone un-tag this post from “vault-release-ce-ent” please, as it does show up in the RSS feeds that everyone monitors for release announcements?