Monitor Telemetry with Prometheus & Grafana

We are trying to monitor Vault using Prometheus in production like in the docs: Monitor Telemetry with Prometheus & Grafana | Vault - HashiCorp Learn
when we create a token it’s expired after 32 days. how do you suggest we will monitor this in production? it will be great if we can have a long-live token.

My approach has been to simply turn on unauthenticated_metrics_access in the Vault config file.

If on production you have sensitive data, and want auth a prometheus scraper, You should implement for example vault agent and get new token from vault.

thank you for the response,
how do you put the new token in Prometheus each time?

We do this in a two step process :

  • On initial deploy, we set a AppRole Role_ID.
  • We have a separate process push new Secret_ID’s ( Single Use - Regular TTL )
  • On Start-up, a separate process ( Vault-Agent or a cron ) will do the AppRole login and get a token. The Token is then continuously renewed ( Tokens | Vault - HashiCorp Learn )

The Point of the separate process pushing a Secret_ID, is for 2 use cases:

  • The token is not renewed for longer than the TTL of the Token.
  • We suspect a breach (Via the audit log monitoring), revoke the token.

If you are using the Vault agent - Prometheus will talk to the LOCAL Vault-agent endpoint. Vault-agent “proxies” the connection to the Vault-server.