Exec driver: memory.kmem.limit_in_bytes: operation not supported

I have a job failing with this error:

2022-02-03T20:51:53-06:00  Driver Failure         failed to launch command with executor: rpc error: code = Unknown desc = container_linux.go:367: starting container process caused: process_linux.go:340: applying cgroup configuration for 
process caused: failed to write "1": write /sys/fs/cgroup/memory/nomad/f4295063-fc79-1e6e-aca7-37708269afcc/memory.kmem.limit_in_bytes: operation not supported

For reference, here is what I see for resource isolation capabilities:

$ awk '{print $1 " " $4}' /proc/cgroups
#subsys_name enabled
cpuset 1
cpu 1
cpuacct 1
blkio 1
memory 1
devices 1
freezer 1
net_cls 1
perf_event 1
net_prio 1
hugetlb 1
pids 1
rdma 1
misc 1

This is with Nomad running as root on openSUSE Leap 15.3:

$ cat /etc/os-release 
NAME="openSUSE Leap"
VERSION="15.3"
ID="opensuse-leap"
ID_LIKE="suse opensuse"
VERSION_ID="15.3"
PRETTY_NAME="openSUSE Leap 15.3"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:leap:15.3"
BUG_REPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org/"

Any tips on what might be causing this issue?

The plot has gotten thicker.

I am able to reproduce it without Nomad by booting a fresh copy of my image, which is based on openSUSE Leap 15.3:

$ cd /sys/fs/cgroup/memory
$ mkdir cg1 && cd cg1
$ echo 1 > memory.kmem.limit_in_bytes

This results in the same “operation not supported” error.

However, when I do the same with a vanilla openSUSE Leap 15.3 image, it seems to work. This seems to indicate that something in my image is somehow causing this to break, which seems odd because it doesn’t do very much other than install a few packages and HashiCorp tools.

Hi @me11. Thanks for using Nomad!

This is very interesting indeed! Do you mind sharing your Dockerfile?