Add additional labels to nomad allocs telemetry

Nomad
1

Hi all,

Is there a way to add additional labels to the Nomad Prometheus metrics?

For instance, it would be helpful if the Nomad meta paremeters were pass through, so that you can do dynamic alert routing in Prometheus/Alertmanager based on said labels per job.

2

Hi @xbglowx,

There is not a way currently to add additional labels to Nomad Prometheus metrics. This is also not supported in the underlying library go-metrics. What information are you looking to provide via labels to perform alerting?

jrasell and the Nomad team.

3

Hi jrasell,

I am looking to have an “owner” meta label passed, so that in my Prometheus alerting rules I can reuse that label to dynamically figure out which alertmanager route to take.

Something like this:

expr: count by (task) (nomad_client_allocs_failed) > 0
labels:
  owner: '{{$labels.owner}}'
annotations:
  description: 'Nomad allocations failures for task {{$labels.task}}'
  summary: 'Nomad Allocations Failures for Task {{$labels.task}}. Count: {{$value}}'