Writing to kvv2 secrets with hcp_root policy is failing with permission denied

I’m using approles for authentication. Here’s the result of approle read:

Key                        Value
---                        -----
bind_secret_id             true
local_secret_ids           false
policies                   [hcp_root]
secret_id_bound_cidrs      <nil>
secret_id_num_uses         100
secret_id_ttl              720h
token_bound_cidrs          []
token_explicit_max_ttl     0s
token_max_ttl              720h
token_no_default_policy    false
token_num_uses             100
token_period               0s
token_policies             [hcp_root]
token_ttl                  720h
token_type                 default

I use secret_id and role_id to create my client in go, which authenticates successfully with the cluster. However, everytime I try to write to the “kv-v2” mountpoint (which I’ve created), it fails with permission denied. Here are some audit logs showing the failure:


{"time":"2024-01-18T03:18:56.443054033Z","type":"request","auth":{"client_token":"hmac-sha256:faea1cb65581a1894758a26b94ac4a3057fea05d9044933e0da10d944526cd5c","accessor":"hmac-sha256:73c288eeed5f99f9b75aa60483ff7ab8b6520e6ec2b31775dffc0200f1f27119","display_name":"admin-auth-approle","policies":["default","hcp_root"],"token_policies":["default","hcp_root"],"policy_results":{"allowed":false},"metadata":{"role_name":"market"},"remaining_uses":100,"entity_id":"b8c2a7b9-d558-717c-1c77-2f87b467b111","token_type":"service","token_ttl":2592000,"token_issue_time":"2024-01-18T03:18:03Z"},"request":{"id":"871e13fa-6a30-26d1-10d2-23423b24782b","client_id":"b8c2a7b9-d558-717c-1c77-2f87b467b111","operation":"create","mount_point":"admin/kv-v2/","mount_type":"kv","mount_accessor":"kv_bf6865cb","mount_running_version":"v0.16.1+builtin","mount_class":"secret","client_token":"hmac-sha256:146eda1e8ca22290882848ff8068145d91ce7147b235892e1cc33de803a3f7e0","client_token_accessor":"hmac-sha256:73c288eeed5f99f9b75aa60483ff7ab8b6520e6ec2b31775dffc0200f1f27119","namespace":{"id":"riVQb","path":"admin/"},"path":"kv-v2/data/kmk/b49bd1e4c8c86314a4beb63d88bb34b1e6b8eff31ca32e6686b620b2d116733d","data":{"data":{"aesKey":"hmac-sha256:a53cfd66aabca76d95ce3934029ec9e0dbb3e5e94a0db98d08e6384207cc41d0"}},"remote_address":"","remote_port":54984},"error":"1 error occurred:\n\t* permission denied\n\n"}
{"time":"2024-01-18T03:18:56.443503868Z","type":"response","auth":{"client_token":"hmac-sha256:faea1cb65581a1894758a26b94ac4a3057fea05d9044933e0da10d944526cd5c","accessor":"hmac-sha256:73c288eeed5f99f9b75aa60483ff7ab8b6520e6ec2b31775dffc0200f1f27119","display_name":"admin-auth-approle","policies":["default","hcp_root"],"token_policies":["default","hcp_root"],"policy_results":{"allowed":false},"metadata":{"role_name":"market"},"remaining_uses":100,"entity_id":"b8c2a7b9-d558-717c-1c77-2f87b467b111","token_type":"service","token_ttl":2592000,"token_issue_time":"2024-01-18T03:18:03Z"},"request":{"id":"871e13fa-6a30-26d1-10d2-23423b24782b","client_id":"b8c2a7b9-d558-717c-1c77-2f87b467b111","operation":"create","mount_point":"admin/kv-v2/","mount_type":"kv","mount_accessor":"kv_bf6865cb","mount_running_version":"v0.16.1+builtin","mount_class":"secret","client_token":"hmac-sha256:146eda1e8ca22290882848ff8068145d91ce7147b235892e1cc33de803a3f7e0","client_token_accessor":"hmac-sha256:73c288eeed5f99f9b75aa60483ff7ab8b6520e6ec2b31775dffc0200f1f27119","namespace":{"id":"riVQb","path":"admin/"},"path":"kv-v2/data/kmk/b49bd1e4c8c86314a4beb63d88bb34b1e6b8eff31ca32e6686b620b2d116733d","data":{"data":{"aesKey":"hmac-sha256:a53cfd66aabca76d95ce3934029ec9e0dbb3e5e94a0db98d08e6384207cc41d0"}},"remote_address":"","remote_port":54984},"response":{"mount_point":"admin/kv-v2/","mount_type":"kv","mount_accessor":"kv_bf6865cb","mount_running_plugin_version":"v0.16.1+builtin","mount_class":"secret","data":{"error":"hmac-sha256:5aee958bb06f61a491bfdab0a77644374fef6454b9deb19032c06096776120c1"}},"error":"1 error occurred:\n\t* permission denied\n\n"}

I’ve removed the remote address from some of the logs.

Not sure where to start looking into the issue, a bit stuck. I’ve verified that approle is enabled, as is kvv2.

Would appreciate any help, thanks!

Typically permission denied is because the call to HCP Vault is not passing the namespace, though I do see the namespace in the audit log errors you shared (not 100% sure if that’s from Vault knowing its in the admin namespace or because youre passing the namespace from your client).

Are you able to authenticate using a simple CLI command or API call using cURL?

I am not a Go developer, so not sure I’d be able to validate your code but might be helpful to see.