Writing to kvv2 secrets with hcp_root policy is failing with permission denied

HashiCorp Cloud Platform (HCP) HCP Vault Secrets
1

I’m using approles for authentication. Here’s the result of approle read:

Key                        Value
---                        -----
bind_secret_id             true
local_secret_ids           false
policies                   [hcp_root]
secret_id_bound_cidrs      <nil>
secret_id_num_uses         100
secret_id_ttl              720h
token_bound_cidrs          []
token_explicit_max_ttl     0s
token_max_ttl              720h
token_no_default_policy    false
token_num_uses             100
token_period               0s
token_policies             [hcp_root]
token_ttl                  720h
token_type                 default

I use secret_id and role_id to create my client in go, which authenticates successfully with the cluster. However, everytime I try to write to the “kv-v2” mountpoint (which I’ve created), it fails with permission denied. Here are some audit logs showing the failure:


{"time":"2024-01-18T03:18:56.443054033Z","type":"request","auth":{"client_token":"hmac-sha256:faea1cb65581a1894758a26b94ac4a3057fea05d9044933e0da10d944526cd5c","accessor":"hmac-sha256:73c288eeed5f99f9b75aa60483ff7ab8b6520e6ec2b31775dffc0200f1f27119","display_name":"admin-auth-approle","policies":["default","hcp_root"],"token_policies":["default","hcp_root"],"policy_results":{"allowed":false},"metadata":{"role_name":"market"},"remaining_uses":100,"entity_id":"b8c2a7b9-d558-717c-1c77-2f87b467b111","token_type":"service","token_ttl":2592000,"token_issue_time":"2024-01-18T03:18:03Z"},"request":{"id":"871e13fa-6a30-26d1-10d2-23423b24782b","client_id":"b8c2a7b9-d558-717c-1c77-2f87b467b111","operation":"create","mount_point":"admin/kv-v2/","mount_type":"kv","mount_accessor":"kv_bf6865cb","mount_running_version":"v0.16.1+builtin","mount_class":"secret","client_token":"hmac-sha256:146eda1e8ca22290882848ff8068145d91ce7147b235892e1cc33de803a3f7e0","client_token_accessor":"hmac-sha256:73c288eeed5f99f9b75aa60483ff7ab8b6520e6ec2b31775dffc0200f1f27119","namespace":{"id":"riVQb","path":"admin/"},"path":"kv-v2/data/kmk/b49bd1e4c8c86314a4beb63d88bb34b1e6b8eff31ca32e6686b620b2d116733d","data":{"data":{"aesKey":"hmac-sha256:a53cfd66aabca76d95ce3934029ec9e0dbb3e5e94a0db98d08e6384207cc41d0"}},"remote_address":"","remote_port":54984},"error":"1 error occurred:\n\t* permission denied\n\n"}
{"time":"2024-01-18T03:18:56.443503868Z","type":"response","auth":{"client_token":"hmac-sha256:faea1cb65581a1894758a26b94ac4a3057fea05d9044933e0da10d944526cd5c","accessor":"hmac-sha256:73c288eeed5f99f9b75aa60483ff7ab8b6520e6ec2b31775dffc0200f1f27119","display_name":"admin-auth-approle","policies":["default","hcp_root"],"token_policies":["default","hcp_root"],"policy_results":{"allowed":false},"metadata":{"role_name":"market"},"remaining_uses":100,"entity_id":"b8c2a7b9-d558-717c-1c77-2f87b467b111","token_type":"service","token_ttl":2592000,"token_issue_time":"2024-01-18T03:18:03Z"},"request":{"id":"871e13fa-6a30-26d1-10d2-23423b24782b","client_id":"b8c2a7b9-d558-717c-1c77-2f87b467b111","operation":"create","mount_point":"admin/kv-v2/","mount_type":"kv","mount_accessor":"kv_bf6865cb","mount_running_version":"v0.16.1+builtin","mount_class":"secret","client_token":"hmac-sha256:146eda1e8ca22290882848ff8068145d91ce7147b235892e1cc33de803a3f7e0","client_token_accessor":"hmac-sha256:73c288eeed5f99f9b75aa60483ff7ab8b6520e6ec2b31775dffc0200f1f27119","namespace":{"id":"riVQb","path":"admin/"},"path":"kv-v2/data/kmk/b49bd1e4c8c86314a4beb63d88bb34b1e6b8eff31ca32e6686b620b2d116733d","data":{"data":{"aesKey":"hmac-sha256:a53cfd66aabca76d95ce3934029ec9e0dbb3e5e94a0db98d08e6384207cc41d0"}},"remote_address":"","remote_port":54984},"response":{"mount_point":"admin/kv-v2/","mount_type":"kv","mount_accessor":"kv_bf6865cb","mount_running_plugin_version":"v0.16.1+builtin","mount_class":"secret","data":{"error":"hmac-sha256:5aee958bb06f61a491bfdab0a77644374fef6454b9deb19032c06096776120c1"}},"error":"1 error occurred:\n\t* permission denied\n\n"}

I’ve removed the remote address from some of the logs.

Not sure where to start looking into the issue, a bit stuck. I’ve verified that approle is enabled, as is kvv2.

Would appreciate any help, thanks!

2

Typically permission denied is because the call to HCP Vault is not passing the namespace, though I do see the namespace in the audit log errors you shared (not 100% sure if that’s from Vault knowing its in the admin namespace or because youre passing the namespace from your client).

Are you able to authenticate using a simple CLI command or API call using cURL?

I am not a Go developer, so not sure I’d be able to validate your code but might be helpful to see.

3

The issue you’re facing with permission denied errors when writing to the kv-v2 secret engine using an approle authenticated with the hcp_root policy likely stems from a mismatch between the policy permissions and the requirements for writing secrets to the kv-v2 engine. Despite having the hcp_root policy attached, ensure that the policy explicitly allows write access to the kv-v2 paths you’re attempting to use. Verify and adjust the policy definitions to include explicit permissions for creating and updating secrets at the specific paths within the kv-v2 engine. Additionally, ensure that the kv-v2 secret engine is correctly mounted and accessible at the paths you’re using.