# Nomad job troubleshooting **URL:** \ **Category:** Nomad\ **Created:** [March 9, 2020, 7:37pm UTC](https://discuss.hashicorp.com/t/nomad-job-troubleshooting/6665 "2020-03-09T19:37:00Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![wstaples](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/wstaples/32/3777_2.png) [@wstaples](https://discuss.hashicorp.com/u/wstaples)\ **Post date:** [March 9, 2020, 7:37pm UTC](https://discuss.hashicorp.com/t/nomad-job-troubleshooting/6665/1 "2020-03-09T19:37:00Z")
Hello, Today I had an issue in which a long running job became stuck at “queued” and “dead”. This issue clearly showed me that I don’t have a lot of experience troubleshooting issues in nomad. I thought I would simply learn as I go but failures in nomad are so rare I haven’t even looked at my jobs in a long time (a year?). The fix for todays issue (which is always how I fix issues) is to use terraform to simply taint the job and re-apply it. (I believe this is the equivalent of stopping and starting a job). Since failures are so few and far between I think I need to put together a generic troubleshooting document. What exactly should I be checking for? In today’s example when I ran “nomad status ”. It had output like: ``` ID = shipping-rates Name = shipping-rates Submit Date = 11/11/19 21:00:08 UTC Type = service Priority = 50 Datacenters = my-data-center Status = dead Periodic = false Parameterized = false Summary Task Group Queued Starting Running Failed Complete Lost shipping-rates 2 0 0 0 0 0 Allocations No allocations placed ``` This is where I got stuck. I did not know what to check after that. I did some googling and found some suggestions with commands including eval and alloc id’s which I did not have. I think I need to find the following information: - Why did nomad change anything? The service was running nomad decided some action needed to be taken and now the service is not running. So question 1 why did nomad decide to change anything? - When did the problem begin? I assume this is directly tied to the first question. When nomad decided to make a change is probably when the issue occurred - Why was the change unsuccessful? In todays example I assume the jobs were stuck at queued until some timeout which lead to the dead status. At this time we have a very small number of jobs that run under nomad. I also have a complete dev environment for trying things out. I look forward to seeing your suggestion on how you troubleshoot your own weird issues. ---
**Author:** ![Wolfsrudel](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/wolfsrudel/32/28465_2.png) [@Wolfsrudel](https://discuss.hashicorp.com/u/Wolfsrudel)\ **Post date:** [March 10, 2020, 12:00am UTC](https://discuss.hashicorp.com/t/nomad-job-troubleshooting/6665/2 "2020-03-10T00:00:49Z")
With your Google research did you come across the following links? > **[Inspect Running Jobs | Nomad - HashiCorp Learn](https://learn.hashicorp.com/nomad/managing-jobs/inspecting-state)** > > Nomad exposes a number of tools and techniques for inspecting a running job. > This is helpful in ensuring the job started successfully. Additionally, it > can output any errors that occurred while starting the job. > **[Access Application Logs for Troubleshooting | Nomad - HashiCorp Learn](https://learn.hashicorp.com/nomad/managing-jobs/accessing-logs)** > > Nomad provides a top-level mechanism for viewing application logs and data > files via the command line interface. This section discusses the nomad alloc > logs command and API interface. ---
**Author:** ![wstaples](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/wstaples/32/3777_2.png) [@wstaples](https://discuss.hashicorp.com/u/wstaples)\ **Post date:** [March 11, 2020, 8:30pm UTC](https://discuss.hashicorp.com/t/nomad-job-troubleshooting/6665/3 "2020-03-11T20:30:34Z")
Thank you those are helpful. ---
**Author:** ![wstaples](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/wstaples/32/3777_2.png) [@wstaples](https://discuss.hashicorp.com/u/wstaples)\ **Post date:** [March 11, 2020, 8:36pm UTC](https://discuss.hashicorp.com/t/nomad-job-troubleshooting/6665/4 "2020-03-11T20:36:27Z")
I’m still struggling to find out what happens. So far I have determined that evaluations are probably what I’m looking for. If I run nomad job status -evals I get a list of eval id’s. When I run nomad eval-status I see some information but no dates so I can’t tell if any of these evals are recent. They all say “TriggeredBy node-update” so that sounds like a start. I tried googling that term but I was not able to find out what that means. ---
**Author:** ![wstaples](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/wstaples/32/3777_2.png) [@wstaples](https://discuss.hashicorp.com/u/wstaples)\ **Post date:** [March 19, 2020, 4:08pm UTC](https://discuss.hashicorp.com/t/nomad-job-troubleshooting/6665/5 "2020-03-19T16:08:12Z")
I have had a second job fail today in the exact same way. As you can see by the output of nomad job status there are no evals or allocations. I don’t know how to get more information without these things. nomad job status -evals -verbose projects-overview ID = projects-overview Name = projects-overview Submit Date = 10/21/19 13:44:48 UTC Type = service Priority = 50 Datacenters = mydc Status = dead Periodic = false Parameterized = false Summary Task Group Queued Starting Running Failed Complete Lost projects-overview 1 0 0 0 0 0 Evaluations ID Priority Triggered By Status Placement Failures Allocations No allocations placed ---
**Author:** ![wstaples](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/wstaples/32/3777_2.png) [@wstaples](https://discuss.hashicorp.com/u/wstaples)\ **Post date:** [March 24, 2020, 3:35pm UTC](https://discuss.hashicorp.com/t/nomad-job-troubleshooting/6665/6 "2020-03-24T15:35:11Z")
I have had another incident today that affected every job on the server. In the nomad logs I found several of these: `worker: dequeued evaluation cb9fb2ae-870c-446e-a98d-b5d27fe421f4` ` [DEBUG] worker: nack for evaluation 6a8e77d4-f097-584c-06cb-760fdad3cf88` When I run `nomad eval-status` I get `No evaluation(s) with prefix or id` I really need to figure out whats going on here. ---
**Author:** ![SunSparc](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/sunsparc/32/1765_2.png) [@SunSparc](https://discuss.hashicorp.com/u/SunSparc)\ **Post date:** [August 27, 2021, 4:54pm UTC](https://discuss.hashicorp.com/t/nomad-job-troubleshooting/6665/7 "2021-08-27T16:54:37Z")
@wstaples, did you ever figure out what was going on? I am struggling to troubleshoot a job also. ---
**Author:** ![gary.bright](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/gary.bright/32/43835_2.png) [@gary.bright](https://discuss.hashicorp.com/u/gary.bright)\ **Post date:** [December 19, 2024, 2:30pm UTC](https://discuss.hashicorp.com/t/nomad-job-troubleshooting/6665/8 "2024-12-19T14:30:12Z")
I would also be grateful if you can share how you knitted these together? my node-update error was due to it patching itself and rebooting at the OS level