Nomad Job Launches UI empty / no history of periodic jobs

Nomad
1

Hello all,

we have a periodic job that runs once a day.
We’d like to check the jobs history in the UI, but its empty

nomad_job_history
nomad_job_history1240×684 32.4 KB

We tried increasing the various garbage collector settings but to no avail.
Any way to get to this info?

regards,
Steffen

2

Hi. As far as I know, i have not found out how to create history of jobs in Nomad. The enterprise Nomad version features audit logs, but i do not think it does the same thing.

The jobs, logs, allocations, deployments, evaluations are garbage collected by Nomad. Yes, you can configure it to an extent.

I have solved it the following way: I have a endless job running that executed nomad operator api /v1/event/stream with good enough token. The logs are saved to a grafana Loki instance, but a file would do just fine. Then I can parse the events and i know everything that has happened in Nomad. Because jsons are newline separated, simple grep | jq usually gives enough infornation, and sonetimes it is easy with a small python script.

3

Thanks for your response. Sounds like a nice idea.

After looking at the nomad configuration file again, I realized that the statements involving garbage collection ended up in the server_join block. I didnt notice that before, nomad did not throw a syntax error. Apparently, you can put whatever you like there.

Anyway, I moved them into the server block, where they belong and everything seems to be working as intended now.

server {
  enabled       = true

  bootstrap_expect = 3
job_gc_threshold = "168h"
eval_gc_threshold = "168h"
batch_eval_gc_threshold = "168h"
deployment_gc_threshold = "168h"

}
1 Like