Nomad stability across unreliable network

Nomad
,
1

Hi everyone,

we are using nomad in our company in an unusual way. We have one server and multiple clients. On windows we don’t run it as service, but basically by calling the nomad.exe directly inside the cmd. This allows to run UI programs via nomad and batch.

Our network is unfortunately not reliable and often the client loses connection to the server and my understanding is, that when this happens, the client will shutdown and stop the program as well. We don’t want that. For us it is fine, if the server is gone, that the client just stays running and reconnects when the server comes back up. We do not mind, if this takes time. We definitely do not want to stop the ui program from running.

I am open for advices. Thanks in advance!

2

Have you read https://www.hashicorp.com/de/blog/managing-applications-at-the-edge-with-hashicorp-nomad and https://developer.hashicorp.com/nomad/tutorials/edge/schedule-edge-services#simulate-client-disconnect? Maybe https://developer.hashicorp.com/nomad/docs/job-specification/group#disconnect is what you are looking for.

1 Like
3

Actually I guess, https://developer.hashicorp.com/nomad/docs/job-specification/disconnect is what you need.

4

FYI, I had similar issues on wifi-connected nodes for Consul. So, I increased the gossip interval:

gossip_lan {
  gossip_interval = "1s"
  probe_interval = "10s"
  probe_timeout = "10s"
}

gossip_wan {
  gossip_interval = "5s"
  probe_interval = "20s"
  probe_timeout = "20s"
}