Vault agent permission denied when performing renew-self operation

Vault
,
1

Hello,

We are trying to render the vault token using the method mentioned here

Also the consul-template is using the vault token generated by the vault agent, when we started using the method mentioned above, the log file of consul-template is flooded with these messages:

[WARN] vault.write(auth/token/create → 7b29c164): renewer done (maybe the lease expired)
[WARN] vault.write(auth/token/create → 7b29c164): failed to renew: Error making API request.
URL: PUT http://127.0.0.1:8201/v1/auth/token/renew-self
Code: 403. Errors:
*1 error occurred:
* permission denied
[WARN] vault.write(auth/token/create → 7b29c164): renewer done (maybe the lease expired)

Though the token is being rendered in the file, the logs are misleading in this case. We are unable to understand which token consul-template is trying to renew or if even consul-template is doing it?

Here is the consul-template config:

consul {
retry {
enabled = true
attempts = 0
backoff = “250ms”
max_backoff = “1m”
}
}
vault {
address = “http://127.0.0.1:8201”
renew_token = false
vault_agent_token_file = “/etc/vault/vault-tokens/vault-token-via-agent”
retry {
enabled = true
attempts = 0
backoff = “250ms”
max_backoff = “1m”
}
}
max_stale = “120s”
log_level = “info”

Thanks!

2

This looks like a Vault permissions issue with the renewing API. I’d suggest, if you haven’t done it, that you first verify that you can do this outside of consul-template. Maybe try with the vault client and go from there.

On a vault-server running in dev mode using the default policy I can run consul-template using the template snippet you included in the Github issue comment.
I can also do it via the command line in the same way using a token with the default policy…

$ vault write /auth/token/create policies=policy_1 no_default_policy=true
Key                  Value
---                  -----
token                s.W2GTzGBPdVgD3tdqSpwqPJiH
token_accessor       b9ZOECumHz3wLVsucYSCXCEs
token_duration       768h
token_renewable      true
token_policies       ["policy_1"]
identity_policies    []
policies             ["policy_1"]

If I set the token to something invalid, then I get an error much like yours…

$ vault write /auth/token/create policies=policy_1 no_default_policy=true
Error writing data to auth/token/create: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/auth/token/create
Code: 403. Errors:

* permission denied

TLDR; double check the token you are using via vault_agent_token_file = “/etc/vault/vault-tokens/vault-token-via-agent” as I think it is missing permissions.

3

Found the issue, when creating the token via consul-template. The policy that is being attached to the token needs these permissions:

{{with secret “/auth/token/create” “policies=policy_1e” “no_default_policy=true”}}
“X_VAULT_TOKEN” = “{{.Auth.ClientToken}}”
{{ end }}

path “auth/token/renew-self” {
capabilities = [“update”]
}

Hope this helps someone!

4 Likes
4

Hi,
I have one question.

I’m working on integrating HashiCorp Vault into our application using Vault Agent for authentication. The initial setup works well, where the application reads the Vault token from a file generated by Vault Agent and uses it to authenticate with the Vault server.

However, I’m concerned about handling scenarios where the token’s max TTL is reached, and a new token is generated by Vault Agent.

Currently, our application reads the token once during initialization and uses it for subsequent operations. If the token expires and a new one is generated, the application wouldn’t automatically know about the new token, which could lead to failed operations.

To address this, I am thinking to implement a file watcher that monitors the token file for changes. When a new token is generated, the watcher reloads the token and updates the Vault client. While this seems to work in theory, I want to ensure that we’re following best practices and not missing any important considerations.

Here are the specific questions I have:

  1. Is monitoring the token file for changes and reloading the token dynamically the recommended approach for handling token renewal with Vault Agent?
  2. Are there any potential pitfalls or edge cases I should be aware of when implementing this solution?
  3. Are there more efficient or reliable methods to ensure the application always has access to a valid token, especially in high-availability or production environments?

I’d appreciate any feedback or suggestions on improving this implementation.