Hello Tech nomad’s
I am struggling to run a sample job using podman driver, whereas my 3 server node cluster and 3 client node cluster is working and healthy, I have installed podman and podmanand its working fine too (I did not install docker)
I have also installed podman driver plugin as per the official document for ubuntu install Drivers: podman | Nomad | HashiCorp Developer
Here are some questions which i couldn’t find the answer from the official docs / internet /gpt’s
-
How to verify / list which all drivers we currently have installed with their versions (commandline / gui)
I tried nomad plugin status on client node which is not showing me podman driver/plugin
-
where i should install podman or any other driver?
a) on single client node
b) all client nodes
c) single server node
d) all sever node
e) everywhere all clients and all sever nodes
-
Is there any additional configuration required in server or client nomad.hcl ? apart from installing the driver ?
-
Do i need to restart the nomad service on client or servers post plugin install?
-
what could have possibly gone wrong in my case
podman version 4.9.3
Nomad v1.8.3
BuildDate 2024-08-13T07:37:30Z
Revision 63b636e5cbaca312cf6ea63e040f445f05f00478
I could able to move forward little bit after adding plugin_dir manually in my client configuration
plugin_dir = "/opt/nomad/data/plugins"
Now i am getting some error logs related to nomad-driver-podman which i call it some progress
I have installed podman via apt package manager in my ubuntu vm
sudo apt search podman | grep installed`
nomad-driver-podman/noble,now 0.6.1-1 arm64 [installed]
podman/noble-updates,noble-security,now 4.9.3+ds1-1ubuntu0.1 arm64 [installed]
podman service is also up and running too, but i am getting to these error when i restarting nomad service
[WARN] agent.plugin_loader: plugin not referenced in the agent configuration file, future versions of Nomad will not load this plugin until the agent configuration is updated: plugin_dir=/opt/nomad/data/plugins plugin=nomad-driver-podman
[INFO] agent: detected plugin: name=podman type=driver plugin_version=0.6.1
[ERROR] client.driver_mgr.nomad-driver-podman: Could not get podman version: driver=podman error="Get \"http://u/libpod/_ping\": dial unix ///run/user/996/podman/podman.sock: connect: no such file or directory" @module=podman timestamp=2024-08-28T19:15:42.721Z
its trying to look for podman.sock inside user 996 whereas i have podman.sock is available at different location
/run/podman/podman.sock
AND
/run/user/1000/podman/podman.sock
how can i change that ?
finally, i gave up with podman 
its taking a lot of efforts but i am reaching nowhere and now i decided to go with happy path with defaults, and i choose to install docker.
Docker is installed successfully and its working without issues, but somehow nomad is not able to communicate with docker too 
Seems like permission issue, please help 
[ERROR] client.driver_mgr.docker: failed to list pause containers for recovery: driver=docker error="Get \"http://unix.sock/containers/json?filters=%7B%22label%22%3A%5B%22com.hashicorp.nomad.alloc_id%22%5D%7D\": dial unix /var/run/docker.sock: connect: permission denied"
with my further investigation, i found that when i am running the nomad in dev mode, it is able to find the drivers and work perfectly fine,
But the problem is only when running it for serious business (cluster mode)
Hi @codekarta,
You should be running Nomad Clients as root user. From the permission denied errors, it l guess you are not doing this.
Nomad clients should be run as root due to the OS isolation mechanisms that require root privileges. While it is possible to run Nomad as an unprivileged user, careful testing must be done to ensure the task drivers and features you use function as expected. The Nomad client’s data directory should be owned by root with filesystem permissions set to 0700 .
ref: Nomad Agent | Nomad | HashiCorp Developer
