Struggling to run nomad as non root user

sassodev · September 25, 2022, 7:43pm

Hey everybody,
I am setting up a vagrant vm to test out nomad, I know about the dev mode, but I wanted to build a mini cluster on my machine.

This machine is a nomad server, to set it up I am following the official documentation.

The problem is that I am struggling to start nomad as nomad user.

This is the systemd output with the failure status=1

 nomad.service - Nomad
     Loaded: loaded (/etc/systemd/system/nomad.service; enabled; vendor preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Sun 2022-09-25 19:22:28 UTC; 1s ago
       Docs: https://www.nomadproject.io/docs/
    Process: 5848 ExecStart=/usr/local/bin/nomad agent -config /etc/nomad.d (code=exited, status=1/FAILURE)
   Main PID: 5848 (code=exited, status=1/FAILURE)
        CPU: 85ms

Journalctl :

Sep 25 19:29:44 server nomad[7030]: ==> Error loading configuration from /etc/nomad.d: open /etc/nomad.d: permission denied
Sep 25 19:29:44 server systemd[1]: nomad.service: Main process exited, code=exited, status=1/FAILURE
Sep 25 19:29:44 server systemd[1]: nomad.service: Failed with result 'exit-code'.

It’s clear this is a permission issue but why I have no clue.

This are the permissions on /etc/nomad.d/

drwx------  2 root root 4096 Sep 25 19:18 .
drwxr-xr-x 73 root root 4096 Sep 25 19:18 ..
-rw-r--r--  1 root root  373 Sep 25 19:18 server.hcl

This is my nomad service file, pretty standard since it’s from the official docs:

[Unit]

Description=Nomad

Documentation=https://www.nomadproject.io/docs/

Wants=network-online.target

After=network-online.target

# When using Nomad with Consul it is not necessary to start Consul first. These

# lines start Consul before Nomad as an optimization to avoid Nomad logging

# that Consul is unavailable at startup.

#Wants=consul.service

#After=consul.service

[Service]

# Nomad server should be run as the nomad user. Nomad clients

# should be run as root

User=nomad

Group=nomad

ExecReload=/bin/kill -HUP $MAINPID

ExecStart=/usr/local/bin/nomad agent -config /etc/nomad.d

KillMode=process

KillSignal=SIGINT

LimitNOFILE=65536

LimitNPROC=infinity

Restart=on-failure

RestartSec=2

[Install]

WantedBy=multi-user.target

I am pretty sure it’s something stupid or embarassing preventing nomad to run as nomad user.
Do you any advice ?
Thanks to anyone kind enough to reply

I did not state it before but if I run nomad as root user, everything is fine.

sassodev · September 25, 2022, 7:54pm

Well, this was embarassing.
Seconds after posting I found a working solution.

nomad user must be owner of /etc/nomad.d and /opt/nomad.
It should have also execute permissions on the configuration file inside /etc/nomad.d/

I will check my bootstrap script.

dbd · September 25, 2022, 7:58pm

Nop, Nomad doesn’t need to be owner of /etc/nomad.d. It just needs read permission (the problem was that /etc/nomad.d was owned by root with 700 permissions). Execute permission on files inside /etc/nomad.d is also not needed

sassodev · September 25, 2022, 8:21pm

So, other users might just need to read files inside /etc/nomad.d/ right?
Well, theese are my permissions now:

drwx---r--  2 root root 4096 Sep 25 20:08 .
drwxr-xr-x 73 root root 4096 Sep 25 20:08 ..
-rw-r--r--  1 root root  373 Sep 25 20:08 server.hcl

everyone should be able to read server.hcl but this is what I get if try to run nomad as nomad user:

root@server:~# runuser -u nomad -- nomad agent -config=/etc/nomad.d/server.hcl
==> Error loading configuration from /etc/nomad.d/server.hcl: stat /etc/nomad.d/server.hcl: permission denied

Then, when I do add execute permession for other users, this is what I get:

==> Error starting agent: setting up server node ID failed: mkdir /opt/nomad/server: permission denied
    2022-09-25T20:16:59.745Z [WARN]  agent.plugin_loader: skipping external plugins since plugin_dir doesn't exist: plugin_dir=/opt/nomad/plugins
    2022-09-25T20:16:59.747Z [INFO]  agent: detected plugin: name=exec type=driver plugin_version=0.1.0
    2022-09-25T20:16:59.747Z [INFO]  agent: detected plugin: name=qemu type=driver plugin_version=0.1.0
    2022-09-25T20:16:59.747Z [INFO]  agent: detected plugin: name=java type=driver plugin_version=0.1.0
    2022-09-25T20:16:59.747Z [INFO]  agent: detected plugin: name=docker type=driver plugin_version=0.1.0
    2022-09-25T20:16:59.747Z [INFO]  agent: detected plugin: name=raw_exec type=driver plugin_version=0.1.0
    2022-09-25T20:16:59.747Z [ERROR] agent: error starting agent: error="setting up server node ID failed: mkdir /opt/nomad/server: permission denied"

root@server:~# ls -la /opt/nomad/
total 8
drwxr-xr-x 2 root root 4096 Sep 25 20:08 .
drwxr-xr-x 4 root root 4096 Sep 25 20:08 ..

This agent is in server mode, shouldn’t /opt/nomad be owned by the nomad user?
I am confused.

Thanks for your reply, any help is very appreciated

dbd · September 25, 2022, 8:34pm

/etc/nomad.d is a directory, so the user running nomad needs both r and x perm (but x is only needed on the parents, not the files inside)

chown :nomad /etc/nomad.d
chmod 750 /etc/nomad.d

This user must also have write access to nomad data directory, so, if it’s /opt/nomad then you need to set the appropriate permissions or ACL on it, for example

chown -R nomad:nomad /opt/nomad

sassodev · September 29, 2022, 6:28am

Thanks a lot man, now everything is working as intended!

ynnckvdv · August 18, 2023, 11:35pm

I ran into this issue and sadly this doesn’t fix it for me. These are my perms:

user@server:~$ sudo ls -la /etc/nomad.d/
total 16
drwx------  2 root root 4096 Aug 19 00:25 .
drwxr-xr-x 70 root root 4096 Aug 19 00:25 ..
-rw-r--r--  1 root root   43 Aug 19 00:25 nomad.hcl
-rw-r--r--  1 root root   51 Aug 19 00:25 server.hcl

user@server:~$ sudo ls -la /etc/nomad.d/
total 16
drwx------  2 root root 4096 Aug 19 00:25 .
drwxr-xr-x 70 root root 4096 Aug 19 00:25 ..
-rw-r--r--  1 root root   43 Aug 19 00:25 nomad.hcl
-rw-r--r--  1 root root   51 Aug 19 00:25 server.hcl

So /etc/nomad.d is owned by the nomad user, it has R and X perm on the dir & it’s the owner of /opt/nomad . Any idea what I’m missing?

dbd · August 19, 2023, 8:29am

Nop, your /etc/nomad.d is not accessible to nomad because it’s root owned with 700 permissions. You can for example change group to nomad and chmod 750 this dir, for the nomad user to be able to read its content

ynnckvdv · August 19, 2023, 9:35am

Right, I didn’t change that back from an earlier try, it seems. I also realised my binary in the unit file was pointing to /usr/local/bin/nomad instead of /usr/bin/nomad, so I changed that. But (!) I indeed didn’t change my /opt/nomad like you said earlier. Now it’s fixed, thanks!

Weird that the official docs mention that /etc/nomad.d needs to be 750, instead of 700 like you say.

dbd · August 19, 2023, 9:48am

It doesn’t “need” to be 700 or 750 specificaly. The user running Nomad just need r and x perm on the dir, and r perm on the files inside.