Exit code definitions (Exit Code: 28)

[chomey]

I’m seeing this in my Nomad UI.

I have no idea where the “28” is coming from.
The logs from Nomad at that time are equally cryptic:

Jan 18 15:50:11 ip-10-200-25-221 nomad[1211]:     2022-01-18T15:50:11.917Z [INFO]  client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=8457845b-7650-5b2e-6470-22e3d3bb14fa task=task @module=logmon path=/opt/nomad/alloc/8457845b-7650-5b2e-6470-22e3d3bb14fa/alloc/logs/.task.stdout.fifo timestamp=2022-01-18T15:50:11.917Z
Jan 18 15:50:11 ip-10-200-25-221 nomad[1211]:     2022-01-18T15:50:11.917Z [INFO]  client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=8457845b-7650-5b2e-6470-22e3d3bb14fa task=task @module=logmon path=/opt/nomad/alloc/8457845b-7650-5b2e-6470-22e3d3bb14fa/alloc/logs/.task.stderr.fifo timestamp=2022-01-18T15:50:11.917Z
Jan 18 15:50:11 ip-10-200-25-221 nomad[1211]:     2022-01-18T15:50:11.929Z [INFO]  client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/opt/nomad/rawExec.sh Args:[*redacted args*]}"
Jan 18 15:52:21 ip-10-200-25-221 nomad[1211]:     2022-01-18T15:52:21.186Z [INFO]  client.alloc_runner.task_runner: not restarting task: alloc_id=8457845b-7650-5b2e-6470-22e3d3bb14fa task=task reason="Policy allows no restarts"
Jan 18 15:52:21 ip-10-200-25-221 nomad[1211]:     2022-01-18T15:52:21.188Z [INFO]  client.gc: marking allocation for GC: alloc_id=8457845b-7650-5b2e-6470-22e3d3bb14fa

The stderr/stdout files from this allocation are also empty:

If I retry the task, it works (the rawExec works, I get more logs from Nomad, and the task.stderr/task.stdout have the expected contents).

I’m at a loss at how to go about debugging/resolving this. Any help would be much appreciated

[jrasell]

Hi @chomey. The 28 exit code would be coming from the application that is being run under Nomad via the raw-exec driver; Nomad bubbles this value up. The logs are likely empty because the underlying application failed before it had a change to perform any log output.

I can’t explain why the task would work OK a second time around with the current information available. Are you able to share the job specification you are using?

Thanks,
jrasell and the Nomad team