Vault agent permission denied when performing renew-self operation

Vault
,
1

Hello,

We are trying to render the vault token using the method mentioned here

Also the consul-template is using the vault token generated by the vault agent, when we started using the method mentioned above, the log file of consul-template is flooded with these messages:

[WARN] vault.write(auth/token/create → 7b29c164): renewer done (maybe the lease expired)
[WARN] vault.write(auth/token/create → 7b29c164): failed to renew: Error making API request.
URL: PUT http://127.0.0.1:8201/v1/auth/token/renew-self
Code: 403. Errors:
*1 error occurred:
* permission denied
[WARN] vault.write(auth/token/create → 7b29c164): renewer done (maybe the lease expired)

Though the token is being rendered in the file, the logs are misleading in this case. We are unable to understand which token consul-template is trying to renew or if even consul-template is doing it?

Here is the consul-template config:

consul {
retry {
enabled = true
attempts = 0
backoff = “250ms”
max_backoff = “1m”
}
}
vault {
address = “http://127.0.0.1:8201”
renew_token = false
vault_agent_token_file = “/etc/vault/vault-tokens/vault-token-via-agent”
retry {
enabled = true
attempts = 0
backoff = “250ms”
max_backoff = “1m”
}
}
max_stale = “120s”
log_level = “info”

Thanks!

2

This looks like a Vault permissions issue with the renewing API. I’d suggest, if you haven’t done it, that you first verify that you can do this outside of consul-template. Maybe try with the vault client and go from there.

On a vault-server running in dev mode using the default policy I can run consul-template using the template snippet you included in the Github issue comment.
I can also do it via the command line in the same way using a token with the default policy…

$ vault write /auth/token/create policies=policy_1 no_default_policy=true
Key                  Value
---                  -----
token                s.W2GTzGBPdVgD3tdqSpwqPJiH
token_accessor       b9ZOECumHz3wLVsucYSCXCEs
token_duration       768h
token_renewable      true
token_policies       ["policy_1"]
identity_policies    []
policies             ["policy_1"]

If I set the token to something invalid, then I get an error much like yours…

$ vault write /auth/token/create policies=policy_1 no_default_policy=true
Error writing data to auth/token/create: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/auth/token/create
Code: 403. Errors:

* permission denied

TLDR; double check the token you are using via vault_agent_token_file = “/etc/vault/vault-tokens/vault-token-via-agent” as I think it is missing permissions.

3

Found the issue, when creating the token via consul-template. The policy that is being attached to the token needs these permissions:

{{with secret “/auth/token/create” “policies=policy_1e” “no_default_policy=true”}}
“X_VAULT_TOKEN” = “{{.Auth.ClientToken}}”
{{ end }}

path “auth/token/renew-self” {
capabilities = [“update”]
}

Hope this helps someone!

4 Likes