Nomad 1.7.3 - upgraded from 1.0.4 - cpu and memory allocation showing as zero

Nomad
1

I have been experimenting with nomad and the raw_exec driver (client running as a non root user) and on version 1.0.4 (integrated with consul) I could see CPU and Memory stats for the allocations. e.g. go to topology, click on a Job and you see the aggregated memory and cpu graphs for that job.

I recently upgraded (OK, removed and re-installed on the same host) using Nomad 1.7.3 and instead of consul for service discovery, I’m using Nomad itself (i.e. not using consul at all).

Again, Nomad clients are running as non-root. Jobs are running (everything working as before), except the cpu and memory allocation is showing as zero. e.g.

nomad alloc status -stats 6738ce16
Task Resources:
CPU        Memory       Disk     Addresses
0/200 MHz  0 B/400 MiB  300 MiB

Memory Stats
RSS  Swap
0 B  0 B

CPU Stats
Percent  System Mode  User Mode
0.00%    0.00%        0.00%

And the CPU / Memory graphs for the job are also zero (inspecting the element on the graph and looking at the JSON response for the metrics - all show zeros too).

cat /etc/redhat-release
CentOS Linux release 7.9.2009 (Core)

1x Server (running as root). 3x Clients running as non-root.

Any suggestions?

nomad-1.7.3
nomad-1.7.32332×1926 353 KB

nomad-1.0.4
nomad-1.0.41942×1958 276 KB

2

What is your kernel version? What cgroups do you have mounted and how? What errors and warnings are there in Nomad client logs? I will suspect that you do not have cgroupsv2 on the host. CentOS 7.9 is pretty old and CentOS non-stream is discontinued.

3

Kernel: 3.10.0-1160.59.1.el7.x86_64 #1 SMP Wed Feb 23 16:47:03 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

No errors in the nomad log. Warning about consul (consul client is running on the host, but Nomad has not been configured to use consul).

It’s cgroups v1:

tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_prio,net_cls)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)

Yes, CentOS 7 is pretty old. I suspect that there have been some code changes in Nomad which now require cgroups v2. Unfortunately, this was for testing a possible deployment to a RHEL7 environment. So could well require an OS upgrade before implementing.