I’ve been working through the tutorial Monitor job service metrics with Prometheus, Grafana, and Consul and have got to the point where Nomad says I have 5 healthy jobs and Consul lists all checks as OK, but when I issue the command “nomad service info grafana” I get the response “Error listing service registrations: No service registrations with prefix “grafana” found”.
While working through the example, I found I needed to start Consul with “-grpc-port 8502” and also create JSON files to issue “consul config write” commands to change the service defaults for the foo, prometheus and grafana jobs’ protocols to “http” since the global default is “tcp’‘, so I have a suspicion that this tutorial has not been reviewed or updated in some time.
Anyway, what am I missing to actually expose or register the services so I can actually experiment with this setup?
Hi @derek.dongray,
It looks like the instruction to run nomad service info grafana is a documentation bug, as Grafana job service is not registered as a Nomad Job.
Looks like you have figured out the rest of the missing pieces like configuring the service protocol and the grpc-port for xDS.
In this tutorial, Grafana is exposed via Consul Ingress Gateaway. You can find this in the ingress gateway job spec (Monitor job service metrics with Prometheus, Grafana, and Consul | Nomad | HashiCorp Developer)
To access Grafana, connect to the host running Ingress Gateway on port 3000. You can find the IP and port using the below commands:
nomad job allocs ingress-controller
- Inspect the allocation to find the URL
nomad alloc status <alloc-id> | grep grafana
I hope this helps.
Thanks. I got connected after looking in Portainer. I’m testing using WSL2 on Windows 11 with ‘mirrored’ network mode, which should mean any ports are exposed externally, but while both 172.26.64.14:3000 and 192.168.1.36:3000 work for a browser running inside WSL2, neither work for an external browser. I assume this is something (ingress controller?) limiting access.
The next issue I’ve hit is that, after installing the recommended envoy clusters dashboard it reports “No Data” even after hitting the ‘foo’ app many times. The Prometheus page 192.168.1.36:8081 similarly reports no data from any query I tried. However, I have found metrics output on both 172.26.64.14:9102/metrics and 192.168.1.36:21102/metrics so, especially after you’ve said that there is at least one documentation bug, I have wondered if there’s something missing or wrong in the actual config.
One concern is that the Prometheus configuration makes no mention of port 9102 which is where I’d expect it to connect.
I’ll continue to see if I can get this to work but would be happy to be pointed to an example that works first time!