Empty memory stats for allocations

Nomad
1

Hello!

I noticed that in one of my clusters which happens to be of RaspberryPi 4’s I don’t see memory stats for any allocation. The chart is empty and using cli shows 0 Bytes too. Any ideas why? am I missing something obvious?

> nomad alloc status -stats 30700f94
...
Memory Stats
Cache  Max Usage  RSS  Swap  Usage
0 B    0 B        0 B  0 B   0 B
...

swappy-20211215_153617

Nomad version 1.2.2
Raspberry PI 4 (arm64)
Ubuntu 20.04.2 LTS

Thanks

2

I have been recently reading about their plugin system. Plugins allow Nomad to see HW on the PC.

I suppose it’s not default naming on the PI and may require some edits to the plugin. I’m not sure where the default location is though sorry.

3

Hi @jcalonso :wave:

Could you check if the Nomad agent is running as sudo? It may not have enough privileges to read system information.

Thanks!

4

Hi @lgfa29
I have Nomad running as root. Since its a homelab setup I have just 3 pi’s and each of them run as server and client (one single systemd unit doing both). Will this be a problem?

5

Might be related, but I think fingerprinting on aarch64 (amd64) has been a bit iffy.

Does the agent report the cpu MHz correctly?

For aarch64, I have this in my code to calculate the total cpu compute for quite some time:

(info farmed from GitHub issues, of course)

current_speed=$(dmidecode -t 4 | grep 'Current Speed:' | awk '{print $3}')
cpu_count=$(cat /proc/cpuinfo | grep 'processor' | wc -l)
CPU_TOTAL_COMPUTE=$(( $current_speed * $cpu_count ))
6

Nope, it shouldn’t be a problem. Client fingerprinting should still work :thinking:

That’s interesting! Did you notice bad fingerprinting in Nomad specifically or just in general?

7

This could be because memory cgroups are not enabled. You could check if you have messages about cgroups in your logs, or try the solution proposed here: kubernetes - Enabling memory cgroup in Ubuntu 20.04 - Ask Ubuntu

8

@shantanugadgil cpu Mhz are fine.

I also noticed that in the client view the memory displays fine, so the issue is just per allocation.

swappy-20211217_172543
swappy-20211217_1725431168×326 18.5 KB

@haimgel
Does this looks ok to you?

grep mem /proc/cgroups
memory	0	62	0
9

The code in my config is based on this:

Last I was wrangling with this was here:

maybe it has been fixed now, I haven’t rechecked.

10

@shantanugadgil nope.
The third column is “enabled”, it’s zero in your case, e.g. memory cgroups are disabled.

11

Thanks @haimgel enabling cgroups did the trick.
This is what I did:

echo -n " cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1" >> /boot/firmware/cmdline.txt
reboot