I have a use case such that my Prometheus server needs a vault token in order to hit the vault metrics endpoint. That token needs to be in the prometheus.yaml file before prometheus starts up.
I’m wondering, is there a way to run the vault agent, and in the template rendering process, can I write out the vault token to the destination file? Is there template syntax that allows me to do that?
I’m working on integrating HashiCorp Vault into our application using Vault Agent for authentication. The initial setup works well, where the application reads the Vault token from a file generated by Vault Agent and uses it to authenticate with the Vault server.
However, I’m concerned about handling scenarios where the token’s max TTL is reached, and a new token is generated by Vault Agent.
Currently, our application reads the token once during initialization and uses it for subsequent operations. If the token expires and a new one is generated, the application wouldn’t automatically know about the new token, which could lead to failed operations.
To address this, I am thinking to implement a file watcher that monitors the token file for changes. When a new token is generated, the watcher reloads the token and updates the Vault client. While this seems to work in theory, I want to ensure that we’re following best practices and not missing any important considerations.
Here are the specific questions I have:
Is monitoring the token file for changes and reloading the token dynamically the recommended approach for handling token renewal with Vault Agent?
Are there any potential pitfalls or edge cases I should be aware of when implementing this solution?
Are there more efficient or reliable methods to ensure the application always has access to a valid token, especially in high-availability or production environments?
I’d appreciate any feedback or suggestions on improving this implementation.