Monitor Telemetry with Prometheus & Grafana

Vault
1

Hello,
We are trying to monitor Vault using Prometheus in production like in the docs: Monitor Telemetry with Prometheus & Grafana | Vault - HashiCorp Learn
when we create a token it’s expired after 32 days. how do you suggest we will monitor this in production? it will be great if we can have a long-live token.
thanks

2

My approach has been to simply turn on unauthenticated_metrics_access in the Vault config file.

3

Hi
If on production you have sensitive data, and want auth a prometheus scraper, You should implement for example vault agent and get new token from vault.

4

thank you for the response,
how do you put the new token in Prometheus each time?

5

We do this in a two step process :

  • On initial deploy, we set a AppRole Role_ID.
  • We have a separate process push new Secret_ID’s ( Single Use - Regular TTL )
  • On Start-up, a separate process ( Vault-Agent or a cron ) will do the AppRole login and get a token. The Token is then continuously renewed ( Tokens | Vault - HashiCorp Learn )

The Point of the separate process pushing a Secret_ID, is for 2 use cases:

  • The token is not renewed for longer than the TTL of the Token.
  • We suspect a breach (Via the audit log monitoring), revoke the token.

If you are using the Vault agent - Prometheus will talk to the LOCAL Vault-agent endpoint. Vault-agent “proxies” the connection to the Vault-server.