Isolated permission errors when using ArgoCD's Vault plugin

sami · June 16, 2022, 2:09am

Hi all,

I’m working to setup ArgoCD to pull secrets out of Hashicorp Vault using ArgoCD’s Vault plugin. Although I am able to read the secrets using the vault CLI in the approle I’ve created I’m having issues requesting secrets back from the Vault using this plugin.

What I’ve done:

I’ve created an approle (argocd) and assigned a policy to it (secret-ro) to ensure that it can read from secrets in the secret/test path.

$ vault read auth/approle/role/argocd
Key                        Value
---                        -----
bind_secret_id             true
local_secret_ids           false
policies                   [secret-ro]
secret_id_bound_cidrs      <nil>
secret_id_num_uses         0
secret_id_ttl              0s
token_bound_cidrs          []
token_explicit_max_ttl     0s
token_max_ttl              0s
token_no_default_policy    false
token_num_uses             0
token_period               0s
token_policies             [secret-ro]
token_ttl                  0s
token_type                 default

$ vault read sys/policy/secret-ro
Key      Value
---      -----
name     secret-ro
rules    path "secret/test" {
   capabilities = ["read", "list"]
}

I’ve created the secret with vault kv put secret/test ingress=my-ingress.domain.com

$ vault read secret/test
Key                 Value
---                 -----
refresh_interval    1h
ingress             my-ingress.domain.com

I’ve entered the role-id and secret-id (as well as the other required variable listed in their docs) into a vars.env file for argocd-vault-plugin to use.
I’ve created an test Kubernetes secret resource with the secret’s path specified in order for the argocd-vault-plugin command to do the replacement
```
kind: Secret
apiVersion: v1
metadata:
  name: example-annotation
type: Opaque
data:
  username: <path:secret/data/test#ingress>
```

Now when I run the cat secret.yaml | argocd-vault-plugin -c vars.env generate - command I expect to have successfully replaced the <path:secret/data/test#ingress> value with the secret stored in the vault. Instead I am getting the following error:

URL: PUT https://redacted.hashicorp.cloud:8200/v1/auth/approle/login
Code: 403. Errors:
* 1 error occurred:
	* permission denied

I have tried a different method of authenticating, by creating the token and utilising that in the argocd-vault-plugin command (as documented here) which is providing a different error message:

Error: Replace: could not replace all placeholders in Template:
Error making API request.

URL: GET https://redacted.hashicorp.cloud:8200/v1/secret/data/test
Code: 403. Errors:

* 1 error occurred:
	* permission denied

So at this point I thought my policy configuration might have been incorrect, so I’ve logged into the vault CLI (vault login) using the same token that argocd-vault-plugin is using and verified that I can read the secret:

$ vault read secret/test
Key                 Value
---                 -----
refresh_interval    1h
ingress             my-ingress.domain.com

Now I’m coming up blank. I’m not sure what else I can look into to try and resolve this issue so any pointers or insight into this would be greatly appreciated. I imagine there’s something small and silly that I’ve missed.

Any thoughts?

sami · June 16, 2022, 4:47am

I have listed the steps taken to attempt to get this working below.

Vault

Enable approle authentication method

vault auth enable approle

Create new approle

$ vault write auth/approle/role/argocd-vault-plugin token_num_uses=0 secret_id_num_uses=0 policies="argocd-vault-plugin-policy"
Success! Data written to: auth/approle/role/argocd-vault-plugin

Get approle role-id and secret-id and store in vars.env file

$ vault read auth/approle/role/argocd-vault-plugin/role-id
Key        Value
---        -----
role_id    role-id-1234

$ vault write -f auth/approle/role/argocd-vault-plugin/secret-id
Key                   Value
---                   -----
secret_id             secret-id-1234
secret_id_accessor    token-accessor-1234
secret_id_ttl         0s

Enable secrets engine at argocd path

$ vault secrets enable -version=2 -path=argocd kv
Success! Enabled the kv secrets engine at: argocd/

Create example secret

$ vault kv put argocd/argocd-vault-plugin/test my-secret=123456789
============ Secret Path ============
argocd/data/argocd-vault-plugin/test

======= Metadata =======
Key                Value
---                -----
created_time       2022-06-16T04:13:11.541399045Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

Create policy for approle to use

$ cat policy.hcl
path "argocd/data/argocd-vault-plugin/test" {
 capabilities = ["read", "list"]
}

$ vault policy write argocd-vault-plugin-policy policy.hcl
Success! Uploaded policy: argocd-vault-plugin-policy

ArgoCD

Write values to vars.env file

$ cat vars.env
VAULT_ADDR="https://public-vault-address.hashicorp.cloud:8200"
AVP_TYPE="vault"
AVP_AUTH_TYPE="approle"
AVP_ROLE_ID="role-id-1234"
AVP_SECRET_ID="secret-id-1234"

Source vars.env file with source vars.env

Write example secret resource

$ cat secret.yaml
kind: Secret
apiVersion: v1
metadata:
  name: example-secret
type: Opaque
data:
  username: <path:argocd/data/argocd-vault-plugin/test#my-secret>

Run argocd-vault-plugin command

Attempted in CLI on local machine

$ cat secret.yaml | argocd-vault-plugin generate -

Error: Error making API request.
URL: PUT https://public-vault-address.hashicorp.cloud:8200/v1/auth/approle/login
Code: 403. Errors:
* 1 error occurred:
    * permission denied

Collect token

$ vault write auth/approle/login role_id="role-id-1234" \
>     secret_id="secret-id-1234"
Key                     Value
---                     -----
token                   token-1234
token_accessor          token-accessor-1234
token_duration          1h
token_renewable         true
token_policies          ["argocd-vault-plugin-policy" "default"]
identity_policies       []
policies                ["argocd-vault-plugin-policy" "default"]
token_meta_role_name    argocd-vault-plugin

Write token to vars.env and change auth type to token

$ cat vars.env
VAULT_ADDR="https://public-vault-address.hashicorp.cloud:8200"
AVP_TYPE="vault"
AVP_AUTH_TYPE="token"
AVP_ROLE_ID="role-id-1234"
AVP_SECRET_ID="secret-id-1234"
VAULT_TOKEN="token-1234"

source vars.env

Run argocd-vault-plugin command

$ cat secret.yaml | argocd-vault-plugin generate -

Error: Replace: could not replace all placeholders in Template:
Error making API request.
URL: GET https://public-vault-address.hashicorp.cloud:8200/v1/argocd/data/argocd-vault-plugin/test
Code: 403. Errors:
* 1 error occurred:
    * permission denied

maxb · June 16, 2022, 6:07am

I have never used HCP Vault - however I understand from other forum posts here, that in HCP Vault, customers don’t have access to the root namespace, and everything happens under a child namespace called admin.

I wonder if your problems are because your manual shell CLI environment is configured for this, but ArgoCD is not.

Beyond that, I’d suggest checking the Vault audit log, to confirm the exact details of the requests that are being denied.

aram · June 16, 2022, 7:32am

I don’t see anything glaring wrong here – you didn’t mention the version of Vault you were using … also are you actually using HCP? You didn’t mention it but your URL look “cloud-ish”, if so you’re missing your namespace parameters everywhere … which would be very odd as your other commands are working.

Two items that I highly recommend using when doing access issues:

A) Turn on your audit device, and check to see what is exactly being accessed and denied. That’ll tell you if your policy and path or your token policy and policy name don’t match.

B) VAULT_TOKEN=<your root or admin token> vault token capabilities <token to test> <direct path>

These will help you diagnose where the issue might be.

So assuming you are usiong the approle token – In your case B should result in:

$ vault token capabilities argocd/data/argocd-vault-plugin/test
list, read

sami · June 21, 2022, 6:37am

Thanks for the replies.

This issue turned out to be a few things.

I had not passed in the namespace that I was using (admin in this case) and the argocd-vault-plugin defaults to root. That solved the main issue.
The ArgoCD version of 2.4.0 has introduced changes to their environment variable handling that breaks the vault plugin. Deploying ArgoCD on 2.4.3 resolved that issue.