Nomad in docker for mac Cgroup segmentation

Nomad
1

Hi,

i am trying to run multiple nomad instances in docker (privileged) on a apple m1 using docker for mac.

if i activate the client, i get a segmentation violation.

i guess this is related to cgroups and resource stanza.

is there any way to simply opt out of this cgroup stuff to make this work?
any alternatives?

nomad_sample-nomad3-1  | ==> Loaded configuration from /nomad_config.hcl
nomad_sample-nomad3-1  | ==> Starting Nomad agent...
nomad_sample-nomad3-1  | panic: runtime error: invalid memory address or nil pointer dereference
nomad_sample-nomad3-1  | [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x1285224]
nomad_sample-nomad3-1  |
nomad_sample-nomad3-1  | goroutine 1 [running]:
nomad_sample-nomad3-1  | github.com/opencontainers/runc/libcontainer/cgroups/fs2.containsDomainController(0x0)
nomad_sample-nomad3-1  | 	github.com/opencontainers/runc@v1.0.3/libcontainer/cgroups/fs2/create.go:68 +0x4
nomad_sample-nomad3-1  | github.com/opencontainers/runc/libcontainer/cgroups/fs2.CreateCgroupPath({0x4000cd4660, 0x1a}, 0x4000cd6e00)
nomad_sample-nomad3-1  | 	github.com/opencontainers/runc@v1.0.3/libcontainer/cgroups/fs2/create.go:118 +0x508
nomad_sample-nomad3-1  | github.com/opencontainers/runc/libcontainer/cgroups/fs2.(*manager).Apply(0x4000014f60, 0xffffffffffffffff)
nomad_sample-nomad3-1  | 	github.com/opencontainers/runc@v1.0.3/libcontainer/cgroups/fs2/fs2.go:71 +0x3c
nomad_sample-nomad3-1  | github.com/hashicorp/nomad/client/lib/cgutil.(*cpusetManagerV2).ensureParent(0x40002817a0)
nomad_sample-nomad3-1  | 	github.com/hashicorp/nomad/client/lib/cgutil/cpuset_manager_v2.go:296 +0x68
nomad_sample-nomad3-1  | github.com/hashicorp/nomad/client/lib/cgutil.(*cpusetManagerV2).Init(0x40002817a0, {0x0, 0x0, 0x0})
nomad_sample-nomad3-1  | 	github.com/hashicorp/nomad/client/lib/cgutil/cpuset_manager_v2.go:71 +0xe4
nomad_sample-nomad3-1  | github.com/hashicorp/nomad/client.(*Client).init(0x4000cb6580)
nomad_sample-nomad3-1  | 	github.com/hashicorp/nomad/client/client.go:689 +0x9a8
nomad_sample-nomad3-1  | github.com/hashicorp/nomad/client.NewClient(0x4000736280, {0x2e25058, 0x4000500008}, {0x2e105c0, 0x4000c8a080}, {0x2e5da78, 0x4000cd0000}, 0x0)
nomad_sample-nomad3-1  | 	github.com/hashicorp/nomad/client/client.go:402 +0x9ec
nomad_sample-nomad3-1  | github.com/hashicorp/nomad/command/agent.(*Agent).setupClient(0x400027e000)
nomad_sample-nomad3-1  | 	github.com/hashicorp/nomad/command/agent/agent.go:930 +0x26c
nomad_sample-nomad3-1  | github.com/hashicorp/nomad/command/agent.NewAgent(0x40008f7800, {0x2eb7100, 0x4000bf9c20}, {0x2e15a20, 0x4000b4a180}, 0x4000b55810)
nomad_sample-nomad3-1  | 	github.com/hashicorp/nomad/command/agent/agent.go:148 +0x25c
nomad_sample-nomad3-1  | github.com/hashicorp/nomad/command/agent.(*Command).setupAgent(0x4000994400, 0x40008f7800, {0x2eb7100, 0x4000bf9c20}, {0x2e15a20, 0x4000b4a180}, 0x4000b55810)
nomad_sample-nomad3-1  | 	github.com/hashicorp/nomad/command/agent/command.go:525 +0x7c
nomad_sample-nomad3-1  | github.com/hashicorp/nomad/command/agent.(*Command).Run(0x4000994400, {0x400004e090, 0x5, 0x5})
nomad_sample-nomad3-1  | 	github.com/hashicorp/nomad/command/agent/command.go:728 +0x67c
nomad_sample-nomad3-1  | github.com/mitchellh/cli.(*CLI).Run(0x400098a3c0)
nomad_sample-nomad3-1  | 	github.com/mitchellh/cli@v1.1.2/cli.go:262 +0x5c4
nomad_sample-nomad3-1  | main.RunCustom({0x400004e080, 0x6, 0x6})
nomad_sample-nomad3-1  | 	github.com/hashicorp/nomad/main.go:115 +0x3d8
nomad_sample-nomad3-1  | main.Run(...)
nomad_sample-nomad3-1  | 	github.com/hashicorp/nomad/main.go:85
nomad_sample-nomad3-1  | main.main()
nomad_sample-nomad3-1  | 	github.com/hashicorp/nomad/main.go:81 +0x70
1 Like
2

Hi @timglabisch there isn’t a way to disable cgroups, but you might have better luck with cgroups v1 - which is activated on operating systems where the v1 hierarchy is present. I’m not familiar with how the docker for mac Linux VM is managed, but basically you’ll want to configure that to run something older like Ubuntu 20.04.

As for this panic, this seems to be a bug in libcontainer. Perhaps this has been fixed already in a more recent version. deps: update opencontainers/runc to v1.1.0 · Issue #12372 · hashicorp/nomad · GitHub

3

same on ubuntu 20.04.

i just wanted to have a dev env to without vagrant, with multiple instances. Seems to be impossible (at least for me) using docker on a m1.

4

I have the same stack trace! But I’m on an Intel Debian Bullseye (Debian 5.10.127-1 (2022-06-30) x86_64). Docker 20.10.17, Nomad 1.3.2.

I don’t really know anything about my Docker container or my Nomad configuration unfortunately.

5

Switching to v1 seems to have worked! I’ve set the systemd.unified_cgroup_hierarchy=0 kernel parameter.