Prometheus doesn't find any Nomad services in tutorial setup

Nomad
1

Having failed to get any metrics read while trying to follow the Monitor job service metrics with Prometheus, Grafana, and Consul tutorial, I decided to try a simpler case with the tutorial https://developer.hashicorp.com/nomad/tutorials/manage-clusters/prometheus-metrics.

I initially got an error indicating that the “task network resources have been deprecated as of Nomad 0.12.0. Please configure networking via group network block” which indicates that this tutorial isn’t maintained either, but moved the network block outside the resources block and got both the fabio and prometheus jobs runnning.

However, when I get to the step of viewing the targets, I get no targets, exactly as in the other tutorial.

image
image507×349 15.6 KB

How do I fix this or is there another working example which shows prometheus monitoring the status of Nomad services?

2

the prometheus logs might tell you why it cant connect. Have you tried connecting to the metrics endpoint of your nomad server?

3

Yes, I did check for output on the metrics endpoint and also tried setting a static target instead of using consul_sd_config, but prabably got that wrong in some way.

As a complete novice with prometheus, where would I find the logs?

4

those would be in the nomad job output