Resources exhausted on fresh deployment

Nomad
3

Hello @Kamilcuk
Thanks for responding. That is weird…

I was not interactively running nomad as root. I tried again as root but received the same results.
I’ve made two changes since my original post.

  1. I downgraded Podman from 5.1.1 to 4.9.3. I built 5.1.1 from the source since v5 is unavailable as a package in Ubuntu 24.04. The same issue is occurring.
  2. On the client nodes, the configuration has been changed to capture logs (hopefully, the log_level directive is what you referred to):
$ cat nomad.hcl
datacenter = "lab"
data_dir   = "/opt/nomad/data"
log_level  = "DEBUG"
plugin_dir = "/opt/nomad/plugins"
plugin "nomad-driver-podman" {
  config {
      socket_path = "unix:///run/podman/podman.sock"
      # Customize other Podman driver plugin options here if needed
  }
}

I can confirm that cgroups v2 are available:

$ cat /sys/fs/cgroup/cgroup.controllers
cpuset cpu io memory hugetlb pids rdma misc

After changing the client configuration file and restarting nomad, I can confirm that it knows about CPU cores. Didn’t see anything about CPU frequency.

$ sudo journalctl -xeu nomad -g "CPU core count"
Jun 30 10:33:16 nomad-client1 nomad[11395]:     2024-06-30T10:33:06.402Z [DEBUG] client.fingerprint_mgr.cpu: detected CPU core count: cores=4

While reviewing logs, I did come across this error:

Jun 30 10:33:16 nomad-client1 nomad[11395]:     2024-06-30T10:33:06.400Z [WARN]  client.fingerprint_mgr: failed to detect bridge kernel module, bridge network mode disabled:
Jun 30 10:33:16 nomad-client1 nomad[11395]:   error=
Jun 30 10:33:16 nomad-client1 nomad[11395]:   | 4 errors occurred:
Jun 30 10:33:16 nomad-client1 nomad[11395]:   | \t* failed to find /sys/module/bridge: stat /sys/module/bridge: no such file or directory
Jun 30 10:33:16 nomad-client1 nomad[11395]:   | \t* module bridge not in /proc/modules
Jun 30 10:33:16 nomad-client1 nomad[11395]:   | \t* module bridge not in /lib/modules/6.8.0-36-generic/modules.builtin
Jun 30 10:33:16 nomad-client1 nomad[11395]:   | \t* module bridge not in /lib/modules/6.8.0-36-generic/modules.dep

I did find a similar discussion here.
I’m unsure if this is a nomad-specific issue, but let me try installing nomad on actual hardware. If the issue persists, I’ll come back here.

Thanks