Nomad template change on Consul update

Nomad
1

This is similar to having a job or task restart/reload when a Vault value changes. However, instead of watching for Vault changes, I have a job that is watching for Consul service catalog changes.

I have run into a situation with a task that has four templates. Each of them has dynamically generated data in it from Consul. When a service is added/removed/healthy/unhealthy etc the templates should re-render. After they are re-rendered, they should send a signal.

The problem I am seeing is that not all of the templates are being updated and not all are sending the signal.

To troubleshoot, I removed the change_mode parameters on all but the main template. After I did this, the task started functioning as expected, minus the dynamic updates from the other templates.

I am guessing that a task may not be able to handle multiple dynamically updating templates, that then also send a signal to the task. This being the case, I may need to combine some of the templates, or perhaps put them in sidecars.

Can anyone else confirm this?

1 Like
2

After further diagnosing, it appears that all the dynamic templates are in fact being being updated. However, the signal that is supposed to be sent to the task is not, but only in certain cases. Two of the templates are dynamically updated by changes in Vault values. The other two templates are dynamically updated by changes in Consul services. All of the templates have the same change_mode and signal settings:

      template {
        change_mode   = "signal"
        change_signal = "SIGUSR2"
        splay         = "2m"
        data ...

If I remove these change_* settings from one of the templates that is updated by changes in Consul services, then the other template sends a signal to the task and the task does a reload.

I am not sure if this is a bug or if it is just a bad idea to have so many templates wanting to send signals for the same task.