Metrics getting generated even for dead nomad allocations

Nomad
1

Hello Team,

We are facing issues with respect to “Default metrics getting generated for Nomad”. These metrics are getting generated even for dead nomad allocations.

We have configured the Nomad default configurations as mentioned on your website.
Link: telemetry Stanza - Agent Configuration | Nomad | HashiCorp Developer

Configurations:
telemetry {
statsd_address = “localhost:8125”
}

&

telemetry {
publish_allocation_metrics = true
publish_node_metrics = true
}

which generating metrics as mentioned in link Metrics Reference | Nomad | HashiCorp Developer

Question: How we can stop collecting metrics data for dead nomad allocations?

Observations:

  1. Even after collectd restart, dead allocations are still pushing metrics data to graphite server from app server.
  2. On app server restart where nomad client and collectd are running, dead allocations stop pushing metrics data to graphite server.
2

Hi @sudhir_hule,

This seems like an unexpected problem and from Nomad’s view allocation metrics should stop being emitted once the allocation has reached a terminal state and the metrics sink has flushed its data.

Are you able to provide examples of the allocation state and corresponding metrics that are being emitted?

If you query the Nomad client directly, do you still see the metrics? This will help rule out collectd causing any problems.

Thanks,
jrasell and the Nomad team