Stopping the Nomad Jobs gracefully

Nomad
1

Hi,

We are using Nomad task’s Java driver for deploying Java spring boot applications. In spring boot application we have added shutdown hook to perform some task before the service is getting stopped. But our observation is that, this shutdown hook is not getting invoked when the job is stopped. It seems Nomad is not performing graceful stop. So wanted to know is there any way we can configure the Nomad task so that jobs can be stopped gracefully. Thanks in advance.

2

Hi @ananthakrishna.udupa,

I’d suggest looking at a combination of task.kill_timeout and kill_signal to control exactly how the task is stopped and killed.

It’s also important to ensure your application is behaving correctly and listening for the correct signals.

Thanks,
jrasell and the Nomad team

3

Thanks jrasell for your quick feedback.

We have already tried setting killSignal as “SIGINT” and “SIGTERM”, and killTimeout as 30s but still shutdown hook is not being invoked.

Thanks,
Ananthakrishna

4

Hi @ananthakrishna.udupa,

Is the application receiving any signals from Nomad? Do you have any logs, a reproduction, or similar to help identify the problem?

The Java driver certainly supports stopping tasks with custom signals so there should be logs or task events to help figure out what is happening.

Thanks,
jrasell and the Nomad team

5

Hi @jrasell ,

I have below Spring-boot application:

image
image806×295 15.4 KB

when I run the app from cmd and do CTRL+C onExit() shutdown hook method will get triggered, please find below screenshot for the same.

image
image1893×624 39.5 KB

And am deploying the same jar in nomad with below job definition.

job “nomad-task” {
datacenters = [“dc1”]
type = “service”
group “caller” {
count = 1
task “api” {
driver = “java”
kill_timeout = 5000000000
kill_signal = “SIGTERM”
config {
jar_path = “C:\practice\nomad-task\target\nomad-task.jar”
jvm_options = ["-Xmx256m", “-Xms128m”]
}
resources {
cpu = 500
memory = 300
network {
port “http” {}
}
}
}
restart {
attempts = 1
}
}
}

when stop the job from nomad, shutdown hook is not getting called, please help me to fix the issue.
nomad.txt (14.3 KB)

1 Like
6

@jrasell ,

Please let me know if any additional details required from my side to understand the issue.

Thanks,
Hareesha S N

7

Hello @jrasell ,

excecutor.out say “sent Ctrl-Break to process” but shutdown hook method not getting called, PFA.

Could you please check this issue.

executor.txt (743 Bytes)

8

Hi @harishsoralamavu,

As per the executor logs it seems the signal is being sent to the task which would indicate Nomad is performing actions as expected. Can you confirm what signals the onExit() function is able to handle?

Thanks,
jrasell and the Nomad team

9

Hi @jrasell ,

It handle CONTROL+C signal.

Thanks,
Hareesha S N

10

Hi @jrasell ,

Ctrl-Break generates a “thread dump” (Control+Break Handler), while Ctrl-C allows my shutdown handler(onExit()) to run.

Please let me know how to send Ctrl-C signal.

Thanks,
Hareesha S N

11

Hi @jrasell ,

Could you please update on this.

Thanks,
Hareesha S N

12

Hi @harishsoralamavu,

As per the Mircosoft docs the signal needed is SIGINT within the kill_signal job specification parameter. If this still fails to work, I would suggest raising a bug report against the Nomad repository with details on the job, logs, and anything else that might be relevant.

Thanks,
jrasell and the Nomad team

13

Any update or findings on this. We ran into the same issue.