Nomad No nodes were eligible for evaluation

andy222 · December 27, 2023, 9:24pm

I have Windows 2019 Core boxes added as agents in my Nomad cluster as seen below:

Nodes that I am targeting also has raw_exec enabled as seen below:

And here is my very simple job with only constraints of kernel to be windows as seen below

job "email-handler-service" {
  datacenters = ["dc1"]
  type = "batch"

  group "email-service" {

    constraint {
    attribute = "${attr.kernel.name}"
    value     = "windows"
  }


   task "raw" {
      driver = "raw_exec"

      artifact {
        source      = "https://myartifcat-repo.com/emailhandler-win.zip"
        destination = "local"
      }

      config {
        command = "dotnet"
        args = ["/local/emailhandler-win/EmailHandler.Service.dll" ,"argument1"]
      }
    }

  }
}

When I plan a job, I get a message:

Task Group “email-service” (failed to place 1 allocation):
* No nodes were eligible for evaluation

Can someone help me remediate the issue. I really like the concept of using Nomad for orchestration of all kind of jobs but I have nothing but struggle running jobs other than docker.

andy222 · December 28, 2023, 4:49pm

Any help here? I am sure I am doing something wrong.

shaun.coss · January 8, 2024, 6:45pm

Did you set the NOMAD_ADDR environment variable from the workstation you ran nomad job run from?

andy222 · January 8, 2024, 9:23pm

I was trying to run jon from one of the nomad client machine only. So i didnt have to set nomad_addr.

nickwales1 · January 11, 2024, 7:20pm

Can you run nomad node status, that should show if the windows clients are available to take workloads.

Also I assume the constraint is correct but you can double check by looking at the client in the UI to see all the constraints available to you including if the kernel.name is indeed windows. I don’t have an immediate way to double check that!

andy222 · January 11, 2024, 9:54pm

I tried nomad node status and all three of my Windows 2019 core boxes shows as eligible and ready to take workloads.

I checked the constraint well… and it is correct… It does exist in property.

Is there anyway, I can see why nomad job plan failed to allocate. If there is a verbose output available or logs somewhere for evaluation… When I am running nomad job plan, it seems I am just shooting in dark.

Here is an output from my evaluation… I came across one thread and executed nomad eval list -json

[
    {
        "AnnotatePlan": false,
        "BlockedEval": "",
        "ClassEligibility": null,
        "CreateIndex": 251442,
        "CreateTime": 1705011579299020994,
        "DeploymentID": "",
        "EscapedComputedClass": false,
        "FailedTGAllocs": {
            "email-service": {
                "AllocationTime": 4613,
                "ClassExhausted": null,
                "ClassFiltered": null,
                "CoalescedFailures": 0,
                "ConstraintFiltered": null,
                "DimensionExhausted": null,
                "NodesAvailable": {},
                "NodesEvaluated": 0,
                "NodesExhausted": 0,
                "NodesFiltered": 0,
                "NodesInPool": 0,
                "QuotaExhausted": null,
                "ResourcesExhausted": null,
                "ScoreMetaData": null,
                "Scores": null
            }
        },
        "ID": "f48e6edd-7d98-5d17-4dbb-5e45cacbba47",
        "JobID": "email-handler-service-win",
        "JobModifyIndex": 251441,
        "ModifyIndex": 251442,
        "ModifyTime": 1705011579299020994,
        "Namespace": "default",
        "NextEval": "",
        "NodeID": "",
        "NodeModifyIndex": 0,
        "PreviousEval": "f5f14ca8-43c3-98f5-95b7-a638bb6ee469",
        "Priority": 50,
        "QueuedAllocations": null,
        "QuotaLimitReached": "",
        "RelatedEvals": null,
        "SnapshotIndex": 251441,
        "Status": "blocked",
        "StatusDescription": "created to place remaining allocations",
        "TriggeredBy": "queued-allocs",
        "Type": "batch",
        "Wait": 0,
        "WaitUntil": null
    },
    {
        "AnnotatePlan": false,
        "BlockedEval": "f48e6edd-7d98-5d17-4dbb-5e45cacbba47",
        "ClassEligibility": null,
        "CreateIndex": 251441,
        "CreateTime": 1705011579295132894,
        "DeploymentID": "",
        "EscapedComputedClass": false,
        "FailedTGAllocs": {
            "email-service": {
                "AllocationTime": 4613,
                "ClassExhausted": null,
                "ClassFiltered": null,
                "CoalescedFailures": 0,
                "ConstraintFiltered": null,
                "DimensionExhausted": null,
                "NodesAvailable": {},
                "NodesEvaluated": 0,
                "NodesExhausted": 0,
                "NodesFiltered": 0,
                "NodesInPool": 0,
                "QuotaExhausted": null,
                "ResourcesExhausted": null,
                "ScoreMetaData": null,
                "Scores": null
            }
        },
        "ID": "f5f14ca8-43c3-98f5-95b7-a638bb6ee469",
        "JobID": "email-handler-service-win",
        "JobModifyIndex": 251441,
        "ModifyIndex": 251443,
        "ModifyTime": 1705011579302070101,
        "Namespace": "default",
        "NextEval": "",
        "NodeID": "",
        "NodeModifyIndex": 0,
        "PreviousEval": "",
        "Priority": 50,
        "QueuedAllocations": {
            "email-service": 1
        },
        "QuotaLimitReached": "",
        "RelatedEvals": null,
        "SnapshotIndex": 251441,
        "Status": "complete",
        "StatusDescription": "",
        "TriggeredBy": "job-register",
        "Type": "batch",
        "Wait": 0,
        "WaitUntil": null
    }
]

It says, none of my nodes are evaluated… that is really strange.

nickwales1 · January 11, 2024, 11:04pm

nomad job run -verbose <path/to/job/file.hcl> provide additional information.

You can also specifically see the evaluation using the ID from that output with:
nomad eval status <eval_id>

I just added a windows machine to my clients with this config file:

data_dir = "C:\\nomad\\data"

name = "windows_box"

plugin "raw_exec" {
  config {
    enabled = true
  }
}

client {
  enabled = true
  server_join {
    retry_join = ["<server_ip>"]
  }
}

and was able to have a job allocated onto it. Would you be able to share your client configuration file?

andy222 · January 12, 2024, 2:52pm

Here is my nomad.hcl from one of my Windows 2019 Servers.

datacenter = "dc1"
data_dir   = "C:/programData/nomad/data"
name       = "wincore21"
bind_addr  = "0.0.0.0"
client {  
  enabled = true
  servers = ["DSCD31:4647", "DSCD32:4647", "DSCD33:4647"]
  node_pool = "windows-2019"
}
server {
  enabled = false
}
advertise {

  http  = "{{ GetInterfaceIP \"Ethernet0\" }}"
  rpc   = "{{ GetInterfaceIP \"Ethernet0\" }}"
  serf  = "{{ GetInterfaceIP \"Ethernet0\" }}"
}

consul {
  address                = "localhost:8500"
  server_service_name    = "nomad"
  client_service_name    = "nomad-client"
  checks_use_advertise   = true
  auto_advertise         = true
  server_auto_join       = true
  client_auto_join       = true
  token                  = "81a18ce5-fab4-9878-4333-dcab532ad93f"
  allow_unauthenticated  = true
  ssl             = false
}

acl {  
enabled = true  
}

plugin "raw_exec" {
  config {
    enabled = true
  }
}

We are 90% windows and .net shop. It is very crucial that we can run Nomad jobs on Windows servers.

Thanks a lot

nickwales1 · January 12, 2024, 3:34pm

Thanks for that, can you try adding the node pool into the job spec, something like this:

job {
  ...
  node_pool = "windows-2019"
  ...

I haven’t actually tried node_pools yet but this seems like it could cause issues with the constraints.

andy222 · January 12, 2024, 3:53pm

I think that was it… It scheduled job on my windows box… It fails… and that is something I can look into but. it did schedule a job on my Windows box. Thanks a ton @nickwales1 for your kind help… I appreciate it.

nickwales1 · January 13, 2024, 7:43pm

That’s great, the next hurdle should be more fun to deal with!
I noticed you were using dotnet in the job spec, there is an IIS task driver that might be of interest!

andy222 · January 16, 2024, 2:59pm

Thanks a lot. We are 100% .net shop and I unfortunately don’t find many examples using Nomad for .net. I will try out IIS driver to schedule .net framework IIS app and see how it goes.

Currently I am struggling to figure out why environment variables are not available in Windows task I am running as follows:

config {
        command = "./local/EmailHandlerService_winx64/EmailHandler.Service.exe"
      }

nickwales1 · January 16, 2024, 3:14pm

Which environment variables specifically? You can specify specific variables with either the env block or by writing a template and enabling the env type.

gladstonesonjustin53 · January 16, 2024, 4:08pm

Exciting news! Signed up for the HashiCorp Developer AI private beta. Looking forward to exploring its capabilities Slide Share. The mention of Nomad adds an extra layer of intrigue. Eager to witness the innovative solutions this developer AI will bring.

andy222 · January 16, 2024, 7:18pm

I am currently getting environment variables from Consul using a template.
I finally made it works with a little tweak…

config {
        command = "powershell.exe"
        args = ["-Command", "Set-Location -Path '.\\local\\EmailHandlerService_winx64';", "& .\\EmailHandler.Service.exe"]
      }

I don’t know what difference it makes… to cd into that directory and then running my exe but it was able to read my environment variables set by Template.

Topic		Replies	Views
Nomad task cant find raw_exec driver on windows Nomad	1	303	July 7, 2022
New User on Windows attempting to run example.nomad Nomad	6	2434	December 17, 2020
Brainstorming A Migration to Nomad; Hoping for Advice Nomad	3	560	August 25, 2021
[Nomad][autoscaler] Worker nodes marked as ineligible even with blocked evaluations Nomad	3	422	October 14, 2022
Simple Nomad Job without Docker or other dependencies Nomad	4	1041	February 24, 2023

Nomad No nodes were eligible for evaluation

Related topics