Prometheus_retention_time explanation

Consul
1

Hello
Im trying to understand how prometheus_retention_time option works.

From docs I understand, that it determines how much time metric value will be stored. If its true what is the sense of recommendation (from docs also) to set it at least 2x prometheus scrape interval.
E.g. prometheus scrape interval is 30s, so prometheus_retention_time = 60s
As a result every second scrape request will get same metric values as first one.

Also it is mentioned in docs that it may have sense to set prometheus_retention_time to high values like 1d or even 31d - what is point of doing this? (having metric with value obsoleted month ago)

2

The metrics system in HashiCorp projects is a poor fit for Prometheus.

It is really built for push-based metrics systems.

The code is structured around “sending” metrics values, and for Prometheus, it is necessary to configure how long the last value sent, sticks around being served to Prometheus scrapes, before it vanishes.

An illustration:

You start up a Consul agent.
It serves a Prometheus metric consul_version{version="...", pre_release="..."} 1 to export metadata to Prometheus about the running version.
After prometheus_retention_time has passed, this metric disappears.

This is unambiguously a bug, to someone who understands how Prometheus metrics are supposed to work.

Filed by me: `consul_version` metric does not work properly with Prometheus · Issue #13498 · hashicorp/consul · GitHub

At one point, I was interested in fixing this, but I wasn’t getting enough engagement from HashiCorp, so I gave up on progressing this.

See also HashiCorp's lack of open-source engagement

3

Thanks for info.
E.g. prometheus_retention_time is 1h and scrape inverval is 1m.
First scrape will get metric value and what will happened on second scrape? Will metric have same value because retention period (1h)? so all scrapes before 1h will get same metric value? or metric will be updated on every scrape?

4

Every time the Vault code produces a new value for a metric, it will update what is exposed to Prometheus scrapes.

The retention time only controls how long metrics remain in the Prometheus scrape, whilst the Vault code is not publishing new values.

1 Like
5

And what interval does consul produce metrics?
Am I understand correctly, that it is Agents - Enable Telemetry Metrics | Consul | HashiCorp Developer
60s for prometheus metrics
Is this interval configurable?

6

No, it’s not done on an interval. Individual metrics are updated whenever the relevant code measures a new value.

1 Like
7

Hello,

Here is a KB article that might help!