Debugging Job Driver Failure


I’m working on getting a minimal, working proof-of-concept Nomad job off the ground. I was able to get the job running within a docker-compose testbed, but I’m now having trouble getting an example running between multiple VMs. I removed my custom job command script and replaced with a simple ‘sleep’ example, but I still run into an error:

Error creating rpc client for executor plugin: plugin exited before we could connect.

I’m not using any custom plugins. Wondering if anyone has run into this issue and has any suggestions on how to further debug. I’m able to ping in both directions between VM’s, so the connection seems fine. Also inspected open ports using lsof -i -P -n and the necessary ports 4646-4648 appear to be in a listening state. Here’s the step by step setup and the log information that I know how to access. Are there any other logs that I can look at?

VM’s: Ubuntu 18.04, 16GB memory
Nomad: v0.8.7

I’m running a non-daemon startup setting. On the server I run:

nomad agent -server -config=/home/user/server.d -bootstrap-expect=1 -bind=

/home/user/server.d contains two files: nomad.hcl and server.hcl:


datacenter = "dc1"
data_dir = "/tmp/nomad"


server {
    enabled = true

    # encrypt gossip communication. 
    encrypt = "I9gHdpTTOsgW7qgHa5WXew=="

On the client, I run:

nomad agent -client -config=/home/user/client.d -bind=

where /home/user/client.d contains the same nomad.hcl as the server as well as the following client.hcl file:

client {
    enabled = true
    options = {
        "driver.raw_exec.enable" = "1"
        "driver.raw_exec.no_cgroups" = "1"

Server log on startup:

==> WARNING: Bootstrap mode enabled! Potentially unsafe operation.
==> Loaded configuration from /home/user/nomad.hcl, /home/user/server.d/server.hcl
==> Starting Nomad agent…
==> Nomad agent configuration:

   Advertise Addrs: HTTP:; RPC:; Serf:
        Bind Addrs: HTTP:; RPC:; Serf:
            Client: false
         Log Level: INFO
            Region: global (DC: dc1)
            Server: true
           Version: 0.8.7

==> Nomad agent started! Log data will stream in below:

2022/02/22 08:46:15 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID: Address:}]
2022/02/22 08:46:15 [INFO] raft: Node at [Follower] entering Follower state (Leader: "")
2022/02/22 08:46:15 [INFO] serf: EventMemberJoin:
2022/02/22 08:46:15.055092 [INFO] nomad: starting 8 scheduling worker(s) for [service batch system _core]
2022/02/22 08:46:15.057395 [ERR] consul: error looking up Nomad servers: server.nomad: unable to query Consul datacenters: Get dial tcp connect: connection refused
2022/02/22 08:46:15.058341 [INFO] nomad: adding server (Addr: (DC: dc1)
2022/02/22 08:46:16 [WARN] raft: Heartbeat timeout from "" reached, starting election
2022/02/22 08:46:16 [INFO] raft: Node at [Candidate] entering Candidate state in term 2
2022/02/22 08:46:16 [INFO] raft: Election won. Tally: 1
2022/02/22 08:46:16 [INFO] raft: Node at [Leader] entering Leader state
2022/02/22 08:46:16.423875 [INFO] nomad: cluster leadership acquired

On the client I connect the following:

nomad node config -update-servers

And I receive the following log message:

2022/02/22 08:49:51.151119 [INFO] client: node registration complete

I have the following job file:

job "testjob1" {
    datacenters = ["dc1"]
    update {
        stagger = "5s"
        max_parallel = 1
    group "testgroup" {
        count = 1
        task "testtask" {
            driver = "raw_exec"

            config {
                command = "/bin/sleep"
                args    = ["600"]

Trying to execute the job from the server using:

nomad job run /path/to/job/job.nomad

Results in:

==> Monitoring evaluation "c87c6010"
    Evaluation triggered by job "testjob1"
    Evaluation within deployment: "e63e4f2f"
    Allocation "931c55b0" created: node "f8812a28", group "testgroup"
    Evaluation status changed: "pending" -> "complete"
==> Evaluation "c87c6010" finished with status "complete"

Checking status:

$ nomad job status

ID        Type     Priority  Status   Submit Date
testjob1  service  50        pending  2022-02-22T08:54:59-08:00

$ nomad job status testjob1

ID            = testjob1
Name          = testjob1
Submit Date   = 2022-02-22T08:54:59-08:00
Type          = service
Priority      = 50
Datacenters   = dc1
Status        = pending
Periodic      = false
Parameterized = false

Task Group  Queued  Starting  Running  Failed  Complete  Lost
testgroup   0       0         0        2       0         0

Future Rescheduling Attempts
Task Group  Eval ID   Eval Time
testgroup   edbb4812  40s from now

Latest Deployment
ID          = e63e4f2f
Status      = running
Description = Deployment is running

Task Group  Desired  Placed  Healthy  Unhealthy  Progress Deadline
testgroup   1        2       0        2          2022-02-22T09:04:59-08:00

ID        Node ID   Task Group  Version  Desired  Status  Created  Modified
8afd80e1  f8812a28  testgroup   0        run      failed  21s ago  19s ago
931c55b0  f8812a28  testgroup   0        run      failed  51s ago  49s ago

Looking at the allocation:

$ nomad alloc status 8afd

ID                   = 8afd80e1
Eval ID              = cf71fe9c
Name                 = testjob1.testgroup[0]
Node ID              = f8812a28
Job ID               = testjob1
Job Version          = 0
Client Status        = failed
Client Description   = <none>
Desired Status       = run
Desired Description  = <none>
Created              = 1m20s ago
Modified             = 1m18s ago
Deployment ID        = e63e4f2f
Deployment Health    = unhealthy
Replacement Alloc ID = 2cc4adb6

Task "testtask" is "dead"
Task Resources
CPU      Memory   Disk     IOPS  Addresses
100 MHz  300 MiB  300 MiB  0     

Task Events:
Started At     = N/A
Finished At    = 2022-02-22T16:55:29Z
Total Restarts = 0
Last Restart   = N/A

Recent Events:
Time                       Type             Description
2022-02-22T08:55:29-08:00  Alloc Unhealthy  Unhealthy because of failed task
2022-02-22T08:55:29-08:00  Not Restarting   Error was unrecoverable
2022-02-22T08:55:29-08:00  Driver Failure   failed to start task "testtask" for alloc "8afd80e1-a12a-29d0-5f2b-f76394d7c94b": error creating rpc client for executor plugin: plugin exited before we could connect
2022-02-22T08:55:29-08:00  Task Setup       Building Task Directory
2022-02-22T08:55:29-08:00  Received         Task received by client