Consul Connect job fails because of $PATH

Nomad
,
1 
envoy_bootstrap: error creating bootstrap configuration for Connect proxy sidecar: exec: "consul": executable file not found in $PATH

Hello I am trying to run a consul connect job on a new host. This same job runs fine on a host where consul is installed in /usr/local/bin/. What I don’t fully understand is how to modify the PATH for Nomad.

I noticed the Limitations in the documentation for service mesh but the root user has consul in the $PATH. Is this a bug? How can I fix?

# echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/bin

# which consul
/opt/bin/consul
2

can you confirm, that the environment variable of the running Nomad binary has /opt/bin/ in it’s PATH ?

ps axwwle would reveal the environment, I’ll leave the grepping upto you.

the PATH you see in the logged in ssh session may not be the same as that of the running process.

3

Thanks for replying. I was able to resolve this by adding an Environment option to my Nomad systemd unit to append /opt/bin/ to the list of paths.

...
[Service]
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/opt/bin/nomad agent -config /etc/nomad.d
Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/bin/"
...

The root cause of this is because I’m using Flatcar Linux to deploy Nomad and /usr is mounted as read-only preventing Nomad from being dropped into any of the existing paths.