I’ve enabled Nomad prometheus metrics with the below config, and I’m finding that I can’t rely on it to set up memory alerts since some metrics are randomly missing. For instance I have the Prometheus ZFS exporter running as a separate nomad job, now I want to see the memory usage of it’s allocations, but the nomad_client_allocs_memory_usage metric does not exist for them. I can find it for many others though. Is this a known issue or is this somehow expected behavior?
telemetry {
collection_interval = "1s"
disable_hostname = true
prometheus_metrics = true
publish_allocation_metrics = true
publish_node_metrics = true
}
Example result of making a request to the metrics endpoint using curl:
❯ curl -sL "http://192.168.10.11:4646/v1/metrics?format=prometheus" | grep zfs
nomad_client_allocs_cpu_allocated{alloc_id="75c1a3a4-b388-225c-5512-4f897cd309c9",job="prom-zfs-exporter",namespace="default",task="prom-zfs-exporter",task_group="zfs-exporter-nucbox2"} 100
nomad_client_allocs_cpu_system{alloc_id="75c1a3a4-b388-225c-5512-4f897cd309c9",job="prom-zfs-exporter",namespace="default",task="prom-zfs-exporter",task_group="zfs-exporter-nucbox2"} 0
nomad_client_allocs_cpu_throttled_periods{alloc_id="75c1a3a4-b388-225c-5512-4f897cd309c9",job="prom-zfs-exporter",namespace="default",task="prom-zfs-exporter",task_group="zfs-exporter-nucbox2"} 0
nomad_client_allocs_cpu_throttled_time{alloc_id="75c1a3a4-b388-225c-5512-4f897cd309c9",job="prom-zfs-exporter",namespace="default",task="prom-zfs-exporter",task_group="zfs-exporter-nucbox2"} 0
nomad_client_allocs_cpu_total_percent{alloc_id="75c1a3a4-b388-225c-5512-4f897cd309c9",job="prom-zfs-exporter",namespace="default",task="prom-zfs-exporter",task_group="zfs-exporter-nucbox2"} 0
nomad_client_allocs_cpu_total_ticks{alloc_id="75c1a3a4-b388-225c-5512-4f897cd309c9",job="prom-zfs-exporter",namespace="default",task="prom-zfs-exporter",task_group="zfs-exporter-nucbox2"} 0
nomad_client_allocs_cpu_total_ticks_count{alloc_id="75c1a3a4-b388-225c-5512-4f897cd309c9",job="prom-zfs-exporter",namespace="default",task="prom-zfs-exporter",task_group="zfs-exporter-nucbox2"} 8203.393548727036
nomad_client_allocs_cpu_user{alloc_id="75c1a3a4-b388-225c-5512-4f897cd309c9",job="prom-zfs-exporter",namespace="default",task="prom-zfs-exporter",task_group="zfs-exporter-nucbox2"} 0
nomad_client_allocs_memory_allocated{alloc_id="75c1a3a4-b388-225c-5512-4f897cd309c9",job="prom-zfs-exporter",namespace="default",task="prom-zfs-exporter",task_group="zfs-exporter-nucbox2"} 2.097152e+07
nomad_client_allocs_memory_rss{alloc_id="75c1a3a4-b388-225c-5512-4f897cd309c9",job="prom-zfs-exporter",namespace="default",task="prom-zfs-exporter",task_group="zfs-exporter-nucbox2"} 1.2537856e+07
nomad_client_allocs_memory_swap{alloc_id="75c1a3a4-b388-225c-5512-4f897cd309c9",job="prom-zfs-exporter",namespace="default",task="prom-zfs-exporter",task_group="zfs-exporter-nucbox2"} 0