When we get the metrics into Prometheus several of the metrics have the datacenter label, however none of them have a label for the region.
So we can build dashboards around Nomad datacenters, but not regions.
Are we missing something? To us it feels very important to be able to group metrics by region and we don’t see a way currently to do that. Well we have a bit of a hack with Grafana variables, regex, and nasty stuff. However we would much prefer metrics actually having the region label.
I believe the expectation here is that the region identifier is left to the scraper in the case of Prometheus. This could be via the job_name or be adding the region identifier within a labels config option.
Are you able to share a snippet of your scrape config or details of how you’re discovering the agents to scrape?
I talked about this internally to make sure my context was correct, and Nomad doesn’t export region as a label because the library we use doesn’t support adding default labels to every metric. Without this, we would need to plumb through the region identifier to every corner of the code which exports metrics.
Apologies for the inconvenience, however, I hope this makes sense.