Nomad task constantly restarts due to uncontrolled template rerender

Nomad
1

Hi folks,

I have the task, which runs DB instance and has the template to ended pg_hba.conf file so that instances can talk to each other via services addresses, published to Consul. It looks something like this

...
      service {
        name = "worker-${NOMAD_ALLOC_INDEX}"
        port = "psql"
      }
...
      template {
        data = <<EOF
{{- range services }}
{{- range service .Name }}
host    all             root              {{ .Address }}/32            trust
{{- end }}
{{- end }}

Now the issue I have with it is that it constantly restarts after start. In allocation events I can see messages again and again:

2022-08-08T11:17:53+03:00  Started           Task started by client
2022-08-08T11:17:53+03:00  Restarting        Task restarting in 0s
2022-08-08T11:17:53+03:00  Terminated        Exit Code: 137, Exit Message: "Docker container exited with non-zero exit code: 137"
2022-08-08T11:17:48+03:00  Restart Signaled  Template with change_mode restart re-rendered
2022-08-08T11:17:47+03:00  Started           Task started by client

How I see this issue:

  1. Task starts and publishes service to Consul
  2. Template engine sees new service in consul so it triggers template update and restarts task
  3. While the task restarts, service gets unpublished from Consul
  4. Template engine sees the list of services updated (service got removed) and triggers template rerender → task restart
  5. While the task restarts service from the previous run gets published into Consul and when the task starts again it sees this update and starts this whole thing again and again.

As a result, I have the task restarting every ~5s

2

Hi @corest. Thanks for using Nomad!

Have you tried setting the change_mode to noop?

3

Hi @DerekStrickland
When I use noop - then if there is new postgres instance added - pg_hba configuration won’t be re-rendered.

4

Actually, it will!. Checkout the docs.

change_mode (string: "restart") - Specifies the behavior Nomad should take if the rendered template changes. Nomad will always write the new contents of the template to the specified destination. The possible values below describe Nomad’s action after writing the template to disk.

5

Well, yes, it will re-render the config.
But can I get it into my container without restarting the container running postgres?

E.g. in template I have

        destination = "local/pg_hba.conf"

and then in container task config

        volumes = [
          "local/pg_hba.conf:/opt/citusdb/pg_hba.conf",
        ]

So I need this behavior of restarting the container on update.
But I somehow need to stop this endless loop of restarts.

For now I solved this issue by moving service from container task into group. This way when container restarts, service doesn’t get deregistered from Consul. But it also means I have unhealthy service in Consul while restarting container