Monitoring registered services with Prometheus

Consul
1

What’s the best way to monitor services registered with Consul using Prometheus? Prometheus primarily uses scraping to export metrics and such however, with something like consul_exporter, I’m able to send service down alerts to alert manager with Prometheus.

I can use Consul health API like http://127.0.0.1:8500/v1/health/state/failing or http://127.0.0.1:8500/v1/health/state/warning to find out whether a service is healthy or not. But there is no Prometheus format for these checks. I’m wondering which alerting mechanism others are using for service down/warning alerts.

I don’t want to use consul_exporter since it has it’s own problems and to me is an additional thing to maintain.

2

Any suggestions about monitoring registered services with Prometheus?

3

Hi,
Prometheus can read the list of services to scrape from Consul see here.
Oliver

4

Not too sure if it’s working …at least with the setup we have. Consul health check passes but Prometheus query “up == 0” says the service(s) is(are) down although in Consul UI they’re all healthy. I’ll check more but not all services are showing as down in Prometheus, just a few.

5

Yeah, there was some scrape issue so it was showing as “down”. I fixed that.

However, now I remember why I asked the original question. How do we monitor when the services have no scrape targets i.e., they don’t expose any metrics.

For example, HTTP Api:
/v1/health/state/warning

There’s no Prometheus format since it is not really a metric from a Prometheus sense. How can we monitor services that do not have scrape targets but are unhealthy and get alerted? The only alternative I found is something called “consul_exporter” but it has it’s own issues.

Thanks

6

Hi,
Your services will need some kind of Prometheus exporter, such as micrometer, so that your services expose metrics on a standard endpoint in Prometheus format.

7

Yes, that is asking for changes to builds and code. Since Consul already has API wondering if we can get that exposed in Prometheus format. Anyway, thanks for the input. We’ll continue to use consul_exporter for now and figure this later.

1 Like
8

+1 to adding service health metrics to Consul builtin telemetry