Nomad periodic job run time mismatch with what has configured in cron

[Roma]

Hi there.

I need nestjs app to be started every minute at hh:mm:00 or couple seconds later. Since there is 5+ seconds required to run container and nestjs app I configured the job to run at hh:mm:55

job "scheduler-publisher" {
  datacenters = ["us-east-1"]
  region      = "global"
  type        = "batch"

  periodic {
    cron             = "55 * * * * * *"
    prohibit_overlap = true
  }
  group "scheduler-publisher" {
    ...
  }
}

My expectation was that the allocation will receive the task at around hh:mm:55 but on some reason it always (every minute) gets the task at ~hh:mm:40.

Another weird thing that the time allocation receives a task is not stable. Initially it was around hh:mm:55. On the next day ~50% of allocations for the periodic job received a task at hh:mm:48 and 50% still at hh:mm:55. Today all allocations receive a task at hh:mm:40.

Not sure that’s expected behaviour. If you know or have any thoughts on why it works like that please share.

Thanks!

[seth.hoenig]

Hi @Roma, generally speaking tasks that need to be executed at a high frequency with precision timing would be best handled by a single always running application using an in-memory timer.

Nomad’s cron scheduling will always be at the mercy of the state of the cluster - it could be the case that resources are not available yet and the task doesn’t get run until resources become available. It could be the case the Nomad scheduler is busy scheduling other jobs for seconds or even minutes before getting to your cronjob. There could be a disruption where Nomad servers need time to elect a new leader, delaying task scheduling.