Nomad client allocation memory stats from telemetry seems confusing

Nomad
1

Hi Community,
I am bit confused with the metrics of an allocation in nomad
The value is shows of nomad_client_allocs_memory_usage for a job differs from what shows in nomad alloc status <allocationID> or the memory stats graph shown in nomad UI.

I refer to this docs Metrics | Nomad by HashiCorp and the metrics in this doc is nomad.client.allocs.memory.usage

nomad version 1.1.2
we scrape telemetry metric via prometheus.

Could anyone explain if nomad.client.allocs.memory.usage and nomad alloc status value of memory as same or which metrics should I looks for exact memory usage of an job/allocation.

Thanks

2

Hi @surajthakur and thanks for the question. To understand better, would you be able to specify what UI and alloc status CLI fields you are referencing?

The nomad.client.allocs.memory.usage metric as you point out is the correct metric to use when understanding the memory usage of an allocation. In the case where there are multiple allocations per job, this metric will be repeated using tags to differentiate between each allocations usage.

Thanks,
jrasell and the Nomad team

3

Thanks @jrasell for the response.

In the output of nomad alloc status 64c696b7-c103-fb64-2e5c-f1134c316c0d

Task Resources
CPU          Memory          Disk     Addresses
373/500 MHz  30 MiB/1.0 GiB  300 MiB

which is similar to in UI

image
image1154×267 22.3 KB

But the metric nomad.client.allocs.memory.usage or what data i get from prometheus stats on scraping

nomad_client_allocs_memory_usage{alloc_id="64c696b7-c103-fb64-2e5c-f1134c316c0d"} --> 939642880

the value from prometheus is in bytes.

So thats the ambiguity/confusion if nomad_client_allocs_memory_usage is same metric shown in nomad alloc status

4

Bumping up this thread.

On Nomad 1.6.1 this disparity still exists.

Using nomad alloc status returns ~322 MiB used.

Task Resources:
CPU         Memory           Disk     Addresses
10/300 MHz  322 MiB/2.2 GiB  300 MiB

But in Datadog, the memory usage shows as >1420 MiB.

Why is this?