The metrics system in HashiCorp projects is a poor fit for Prometheus.
It is really built for push-based metrics systems.
The code is structured around “sending” metrics values, and for Prometheus, it is necessary to configure how long the last value sent, sticks around being served to Prometheus scrapes, before it vanishes.
An illustration:
You start up a Consul agent.
It serves a Prometheus metric consul_version{version="...", pre_release="..."} 1 to export metadata to Prometheus about the running version.
After prometheus_retention_time has passed, this metric disappears.
This is unambiguously a bug, to someone who understands how Prometheus metrics are supposed to work.
Filed by me: `consul_version` metric does not work properly with Prometheus · Issue #13498 · hashicorp/consul · GitHub
At one point, I was interested in fixing this, but I wasn’t getting enough engagement from HashiCorp, so I gave up on progressing this.