Vault Agent with AppRole gets "permission denied" on secrets

Vault
1

I’ve got the following setup:

vault auth enable approle
vault write auth/approle/role/copda policies="copda-static,copda-dynamic"

vault secrets enable -path=copda/dynamic database
vault write copda/dynamic/roles/readonly db_name=postgresql \
	creation_statements=@dynamic-secrets/readonly.sql \
	default_ttl=1h max_ttl=24h
vault policy write copda-dynamic dynamic-secrets/policy.hcl

policy.hcl

path "copda/dynamic/creds/readonly" {
  capabilities = ["read"]
}

(“copda-static” nearly the same, so I’m not posting it)

I can login and read the credentials manually using

vault write auth/approle/login role_id=<role.id> secret_id=<secret.id>
VAULT_TOKEN=<token> vault read copda/dynamic/creds/readonly

but the vault agent is only logging (using the same role id and secret id):

2019/11/20 09:39:14.961767 [WARN] vault.read(copda/static/static-creds/copda): failed to check if copda/static/static-creds/copda is KVv2, assume not: Error making API request.

URL: GET http://127.0.0.1:8200/v1/sys/internal/ui/mounts/copda/static/static-creds/copda
Code: 403. Errors:

* preflight capability check returned 403, please ensure client's policies grant access to path "copda/static/static-creds/copda/"
2019/11/20 09:39:14.963646 [WARN] vault.read(copda/dynamic/creds/readonly): failed to check if copda/dynamic/creds/readonly is KVv2, assume not: Error making API request.

and stopping after x trys.

The template the agent is using is

# dynamic secrets
{{- with secret "copda/dynamic/creds/readonly"}}
localhost:5432:postgres:{{ .data.username }}:{{ .data.password }}
{{- end}}

# static secrets
{{- with secret "copda/static/static-creds/copda"}}
localhost:5432:postgres:copda-static:{{ .data.password }}
{{- end}}
2

Using the sink file generated by the agent:

VAULT_TOKEN=$( cat sink ) vault read copda/static/static-creds/copda
Key                    Value
---                    -----
last_vault_rotation    2019-11-21T09:14:38.854890177+01:00
password               A1a-cH7bNY0wCWwYjLx6
rotation_period        24h
ttl                    23h57m11s
username               copda-static

VAULT_TOKEN=$( cat sink ) vault read -output-curl-string copda/static/static-creds/copda                                                                                                                                     1 ↵
curl -H "X-Vault-Request: true" -H "X-Vault-Token: $(vault print token)" http://127.0.0.1:8200/v1/copda/static/static-creds/copda

curl -H "X-Vault-Request: true" -H "X-Vault-Token: $(vault print token)" http://127.0.0.1:8200/v1/copda/static/static-creds/copda
{"request_id":"022601e5-52c8-4a7f-89e9-ad0a1d7872d5","lease_id":"","renewable":false,"lease_duration":0,"data":{"last_vault_rotation":"2019-11-21T09:14:38.854890177+01:00","password":"A1a-cH7bNY0wCWwYjLx6","rotation_period":86400,"ttl":86210,"username":"copda-static"},"wrap_info":null,"warnings":null,"auth":null}

agent.log

2019/11/21 08:14:34.970434 [WARN] (view) vault.read(copda/static/static-creds/copda): vault.read(copda/static/static-creds/copda): Error making API request.

URL: GET http://127.0.0.1:8200/v1/copda/static/static-creds/copda
Code: 403. Errors:

* 1 error occurred:
        * permission denied

 (retry attempt 5 after "4s")
3

You can ignore this issue. Now it’s working - meaning rendering the template - but still complaining about permissions.

2019/11/21 08:59:16.509199 [WARN] (view) vault.read(copda/static/static-creds/copda): vault.read(copda/static/static-creds/copda): Error making API request.

URL: GET http://127.0.0.1:8200/v1/copda/static/static-creds/copda
Code: 403. Errors:

* 1 error occurred:
        * permission denied

 (retry attempt 5 after "4s")
2019/11/21 08:59:20.645745 [WARN] (runner) could not backup "./.pgpass": link ./.pgpass ./.pgpass.bak: no such file or directory
2019/11/21 08:59:20.645794 [INFO] (runner) rendered "./pgpass.tpl" => "./.pgpass"
4

Hi,
I have some doubt.

I’m working on integrating HashiCorp Vault into our application using Vault Agent for authentication. The initial setup works well, where the application reads the Vault token from a file generated by Vault Agent and uses it to authenticate with the Vault server.

However, I’m concerned about handling scenarios where the token’s max TTL is reached, and a new token is generated by Vault Agent.

Currently, our application reads the token once during initialization and uses it for subsequent operations. If the token expires and a new one is generated, the application wouldn’t automatically know about the new token, which could lead to failed operations.

To address this, I am thinking to implement a file watcher that monitors the token file for changes. When a new token is generated, the watcher reloads the token and updates the Vault client. While this seems to work in theory, I want to ensure that we’re following best practices and not missing any important considerations.

Here are the specific questions I have:

  1. Is monitoring the token file for changes and reloading the token dynamically the recommended approach for handling token renewal with Vault Agent?
  2. Are there any potential pitfalls or edge cases I should be aware of when implementing this solution?
  3. Are there more efficient or reliable methods to ensure the application always has access to a valid token, especially in high-availability or production environments?

I’d appreciate any feedback or suggestions on improving this implementation.

5

If your application is using the vault token, you can test to see when it will expire and start reading as its expiration approaches. This is also the behavior that Vault-Agent uses - depending on the auth method, it will start the renewal / regeneration process at 2/3 rd the TTL. This will also allow you to do a smooth transition as both token will be valid for a period of time.

Your edge case is if the token is revoked or expires and doe not get renewed ( if the secret ID expires or is also revoked. )