I’ve got the exec2 plugin installed and recognized as such on my client node, but job runs fail with a driver failure error.
I see this from journalctl -u nomad.service:
-client nomad[3586]: 2024-08-29T15:55:22.123-0400 [ERROR] agent.plugin_loader: error encountered while scanning stdout: plugin_dir=/opt/nomad/data/plugins error="read |0: file already closed"
Aug 29 15:55:32 nomad-client nomad[3586]: 2024-08-29T15:55:22.123-0400 [INFO] agent.plugin_loader: plugin process exited: plugin_dir=/opt/nomad/data/plugins plugin=/opt/nomad/data/plugins/nomad-driver-exec2 id=3595
Aug 29 15:55:32 nomad-client nomad[3586]: 2024-08-29T15:55:22.137-0400 [ERROR] agent.plugin_loader: error encountered while scanning stdout: plugin_dir=/opt/nomad/data/plugins error="read |0: file already closed"
Aug 29 15:55:32 nomad-client nomad[3586]: 2024-08-29T15:55:22.137-0400 [INFO] agent.plugin_loader: plugin process exited: plugin_dir=/opt/nomad/data/plugins plugin=/opt/nomad/data/plugins/nomad-driver-exec2 id=3601
Aug 29 15:55:32 nomad-client nomad[3586]: 2024-08-29T15:55:22.137-0400 [INFO] agent: detected plugin: name=docker type=driver plugin_version=0.1.0
Aug 29 15:55:32 nomad-client nomad[3586]: 2024-08-29T15:55:22.137-0400 [INFO] agent: detected plugin: name=exec2 type=driver plugin_version=v2.0.0
And the web UI for the client node is showing a number of these client events:
Aug 29, '24 17:40:06 -0400 Driver exec2 Healthy
Aug 29, '24 17:40:06 -0400 Driver exec2 failed to fingerprint driver
Is this an error with the install?
Hi @jsleeuw! Not sure what’s going on given the limited information produced in the logs. Can you post more complete information about your system / environment? It’s easiest to debug things if we can reproduce them.
Thanks Seth, of course, let me provide that.
I’m setting up a local lab environment on my (Apple Silicon) Mac. I’ve used Multipass to provision nomad client and server vms running Ubuntu 24.04. Nomad version is 1.8.3, revision 63b636e5cbaca312cf6ea63e040f445f05f00478.
Here’s the client config:
data_dir = "/opt/nomad/data"
bind_addr = "0.0.0.0"
client {
enabled = true
servers = ["...."]
}
plugin "nomad-driver-exec2" {
config {
unveil_defaults = true
unveil_paths = []
unveil_by_task = true
}
}
Here’s some output from journalctl -u nomad
Aug 30 15:20:24 nomad-client nomad[16476]: 2024-08-30T15:20:24.027-0400 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=b7ee0221-ca3e-8efa-3c01-08b7a9c471a6 task=example type=Received msg=“Task received by client” failed=false
Aug 30 15:20:24 nomad-client nomad[16476]: 2024-08-30T15:20:24.040-0400 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=b7ee0221-ca3e-8efa-3c01-08b7a9c471a6 task=example type=“Task Setup” msg=“Building Task Directory” failed=false
Aug 30 15:20:24 nomad-client nomad[16476]: 2024-08-30T15:20:24.258-0400 [ERROR] client.driver_mgr: error encountered while scanning stdout: driver=exec2 error=“read |0: file already closed”
Aug 30 15:20:24 nomad-client nomad[16476]: 2024-08-30T15:20:24.259-0400 [ERROR] client.driver_mgr: plugin process exited: driver=exec2 plugin=/opt/nomad/data/plugins/nomad-driver-exec2 id=16567 error=“exit status 2”
Aug 30 15:20:24 nomad-client nomad[16476]: 2024-08-30T15:20:24.259-0400 [WARN] client.driver_mgr: failed to reattach to plugin, starting new instance: driver=exec2 error=“singleton plugin exited”
Aug 30 15:20:24 nomad-client nomad[16476]: 2024-08-30T15:20:24.259-0400 [INFO] client.alloc_runner.task_runner: failed to start task because plugin shutdown unexpectedly; attempting to recover: alloc_id=b7ee0221-ca3e-8efa-3c01-08b7a9c471a6 task=example
Aug 30 15:20:24 nomad-client nomad[16476]: 2024-08-30T15:20:24.259-0400 [WARN] client.driver_mgr: received fingerprint error from driver: driver=exec2 error=“plugin is shut down”
Aug 30 15:20:24 nomad-client nomad[16476]: 2024-08-30T15:20:24.281-0400 [ERROR] client.driver_mgr: plugin process exited: driver=exec2 plugin=/opt/nomad/data/plugins/nomad-driver-exec2 id=16714 error=“exit status 2”
Aug 30 15:20:24 nomad-client nomad[16476]: 2024-08-30T15:20:24.282-0400 [WARN] client.driver_mgr: received fingerprint error from driver: driver=exec2 error=“plugin is shut down”
Aug 30 15:20:24 nomad-client nomad[16476]: 2024-08-30T15:20:24.282-0400 [WARN] client.driver_mgr: failed to reattach to plugin, starting new instance: driver=exec2 error=“singleton plugin exited”
Aug 30 15:20:24 nomad-client nomad[16476]: 2024-08-30T15:20:24.283-0400 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=b7ee0221-ca3e-8efa-3c01-08b7a9c471a6 task=example type=“Driver Failure” msg=“failed to start task after driver exited unexpectedly: plugin is shut down” failed=false
Aug 30 15:20:24 nomad-client nomad[16476]: 2024-08-30T15:20:24.284-0400 [ERROR] client.alloc_runner.task_runner: running driver failed: alloc_id=b7ee0221-ca3e-8efa-3c01-08b7a9c471a6 task=example error=“failed to start task after driver exited unexpectedly: plugin is shut down”
Aug 30 15:20:24 nomad-client nomad[16476]: 2024-08-30T15:20:24.284-0400 [INFO] client.alloc_runner.task_runner: not restarting task: alloc_id=b7ee0221-ca3e-8efa-3c01-08b7a9c471a6 task=example reason=“Error was unrecoverable”
Aug 30 15:20:24 nomad-client nomad[16476]: 2024-08-30T15:20:24.284-0400 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=b7ee0221-ca3e-8efa-3c01-08b7a9c471a6 task=example type=“Not Restarting” msg=“Error was unrecoverable” failed=true
Aug 30 15:20:28 nomad-client nomad[16476]: 2024-08-30T15:20:28.347-0400 [WARN] client.alloc_runner.task_runner.task_hook.logmon.nomad: timed out waiting for read-side of process output pipe to close: alloc_id=b7ee0221-ca3e-8efa-3c01-08b7a9c471a6 task=example @module=logmon timestamp=2024-08-30T15:20:28.342-0400
Aug 30 15:20:28 nomad-client nomad[16476]: 2024-08-30T15:20:28.347-0400 [WARN] client.alloc_runner.task_runner.task_hook.logmon.nomad: timed out waiting for read-side of process output pipe to close: alloc_id=b7ee0221-ca3e-8efa-3c01-08b7a9c471a6 task=example @module=logmon timestamp=2024-08-30T15:20:28.346-0400
Aug 30 15:20:28 nomad-client nomad[16476]: 2024-08-30T15:20:28.353-0400 [INFO] client.alloc_runner.task_runner.task_hook.logmon: plugin process exited: alloc_id=b7ee0221-ca3e-8efa-3c01-08b7a9c471a6 task=example plugin=/usr/bin/nomad id=16704
Aug 30 15:20:28 nomad-client nomad[16476]: 2024-08-30T15:20:28.355-0400 [INFO] client.gc: marking allocation for GC: alloc_id=b7ee0221-ca3e-8efa-3c01-08b7a9c471a6
Aug 30 15:20:28 nomad-client nomad[16476]: 2024-08-30T15:20:28.561-0400 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=e34952d2-b51a-4c43-6e2f-c1c40c38fd5a task=example type=Received msg=“Task received by client” failed=false
Aug 30 15:20:28 nomad-client nomad[16476]: 2024-08-30T15:20:28.568-0400 [INFO] client.alloc_runner.task_runner: Task event: alloc_id=e34952d2-b51a-4c43-6e2f-c1c40c38fd5a task=example type=“Task Setup” msg=“Building Task Directory” failed=false
Aug 30 15:20:28 nomad-client nomad[16476]: 2024-08-30T15:20:28.697-0400 [ERROR] client.driver_mgr: plugin process exited: driver=exec2 plugin=/opt/nomad/data/plugins/nomad-driver-exec2 id=16720 error=“exit status 2”
Aug 30 15:20:28 nomad-client nomad[16476]: 2024-08-30T15:20:28.698-0400 [INFO] client.alloc_runner.task_runner: failed to start task because plugin shutdown unexpectedly; attempting to recover: alloc_id=e34952d2-b51a-4c43-6e2f-c1c40c38fd5a task=example
Aug 30 15:20:28 nomad-client nomad[16476]: 2024-08-30T15:20:28.698-0400 [WARN] client.driver_mgr: failed to reattach to plugin, starting new instance: driver=exec2 error=“singleton plugin exited”
Aug 30 15:20:28 nomad-client nomad[16476]: 2024-08-30T15:20:28.699-0400 [WARN] client.driver_mgr: received fingerprint error from driver: driver=exec2 error=“plugin is shut down”
Aug 30 15:20:28 nomad-client nomad[16476]: 2024-08-30T15:20:28.713-0400 [ERROR] client.driver_mgr: plugin process exited: driver=exec2 plugin=/opt/nomad/data/plugins/nomad-driver-exec2 id=16735 error=“exit status 2”
Aug 30 15:20:28 nomad-client nomad[16476]: 2024-08-30T15:20:28.713-0400 [WARN] client.driver_mgr: failed to reattach to plugin, starting new instance: driver=exec2 error=“singleton plugin exited”