Nomad 1.7.x - problems with docker driver

Nomad
1

Hello everyone! After upgrading nomad agent from 1.6.x to 1.7x (tried all of the minor versions of 1.7) jobs, that using docker driver are begin to fail with constraint ‘Missing drivers’. Node status page shows that it doesn’t see docker driver. This happens immediately after upgrading from 1.6 to 1.7 and restarting the nomad agent.

Screenshot 2023-12-21 at 13.38.07
Screenshot 2023-12-21 at 13.38.073154×860 119 KB

Restarting agent with log_level=DEBUG shows not much more info regarding docker driver.

Dec 21 11:57:12 stg-pop-mt-02-aws-se-app-10-11-1-41 nomad[22318]:     2023-12-21T11:57:12.601Z [INFO]  agent: detected plugin: name=docker type=driver plugin_version=0.1.0
Dec 21 11:57:12 stg-pop-mt-02-aws-se-app-10-11-1-41 nomad[22318]:     2023-12-21T11:57:12.716Z [DEBUG] client.driver_mgr.docker: using client connection initialized from environment: driver=docker
Dec 21 11:57:12 stg-pop-mt-02-aws-se-app-10-11-1-41 nomad[22318]:     2023-12-21T11:57:12.719Z [DEBUG] client.driver_mgr: initial driver fingerprint: driver=docker health=undetected description="Driver must run as root"

And after rolling back agent version to 1.6.x everything starting to working again. Maybe there is some breaking changes in agent config introduced in 1.7 that i missed?

Running on Ubuntu 20.04.6, Docker version 20.10.18, build b40c2f6

1 Like
2

I am seeing the same thing. In v1.7.2 the client never “sees” the docker driver. When I roll back to v1.6.5 the docker drive is there, as expected. No changes to config files or anything besides the nomad binary.

Ubuntu 20
Docker version 24.0.7, build afdd53b

3 Likes
3

Is nomad running as the same user in both instances? I’ve adopted the bad practice of running nomad as root on Ubuntu 22 to minimize disappearing docker.

4

Nomad is running under its own user, which is member of “docker” group. But that’s not the case here. Problem somewhere in the area of “missing driver” rather than “access to docker denied”

5

Yeah, same here. I’m running nomad under a “nomad” OS user that is part of the docker group.

I’ve even tried adding the “nomad” OS user to the “root” group and adding sudoer privileges (part of a “throw everything and the kitchen sink at it” session while troubleshooting). 1.7.2 never “saw” the docker driver. Swapping out the nomad binary to 1.6.5 resulted in the docker driver showing up as expected.

1 Like
6

Hi nomad 1.7 requires root for docker driver. It literally has if not root; then no docker; fi logic. The official documentation states that nomad client should be run as root and that running Nomad clent as non root is not supported.

We are running Nomad exclusively as non root user. What I did is I patched nomad to remove the logic and compiled myself. I posted modification for nomad 1.6.3 here Cannot Run Docker Driver as non-root user on system with cgroup v2, but this is not reflected in docs · Issue #17816 · hashicorp/nomad · GitHub , for nomad 1.7.x just disable this if nomad/drivers/docker/fingerprint.go at dd15bdff9c35be3fe2d0c470feea016967acea9d · hashicorp/nomad · GitHub .

I can’t say for Nomad 1.7.2, I am using patched 1.6.3 for some time now and works ok. I will migrate to 1.7.x slowly probably February if its stable, and could post an update then.

Add a like to Rootless Nomad · Issue #13669 · hashicorp/nomad · GitHub .

7

I’ve adopted the bad practice of running nomad as root on

Hi. Running Nomad as root is a good practice, the required, supported and recommended practice. To manage numa, cgroups, cores, run and kill processes of any user id, manage files permissions in jobs allocation directories, handle secret and temporary mount points, Nomad client just has to run as root (or with cap admin, effectively root). This is the usual practice with orchestrations, it’s normal.