Nomad kills tasks with templates using consul kv data

Nomad
1

does anyone have the same problem ?

Nomad version

Nomad v1.3.1 (2b054e38e91af964d1235faa98c286ca3f527e56)

Operating system and Environment details

Linux 5.10.0-14-amd64 #1 SMP Debian 5.10.113-1 (2022-04-29) x86_64 GNU/Linux

Issue

nomad kills tasks with templates using consul kv data

Reproduction steps

we have a nomad job file which is using this template:

template {
                data = <<EOF
{{key "netbox/config/extra.py"}}
EOF
                destination = "local/netbox_configs/configuration/extra.py"
                change_mode   = "signal"
                change_signal = "SIGHUP"
                wait {
                    min     = "5s"
                    max     = "12s"
                }
            }

nomad kills the job from time regularly
and in the nomad logs we find something like this:

Killing 	Template failed: kv.block(netbox/config/extra.py): Unexpected response code: 500 (rpc error making call: i/o deadline reached)

the consul kv store is working normally - which i/o deadline or 500 response does this reference?

if we remove this template and use an artifact from git or an inline template everthing works normally and the job will not be killed at all.

these logfile lines appear on the nomad clients:

[WARN]  agent: (view) kv.block(netbox/config/extra.py): Unexpected response code: 500 (rpc error making call: i/o deadline reached) (retry attempt 1 after \"250ms\")"

"agent: (view) kv.block(netbox/config/extra.py): Unexpected response code: 500 (rpc error making call: i/o deadline reached) (retry attempt 1 after \"250ms\")"

[DEBUG] agent: (runner) missing dependency: kv.block(netbox/config/extra.py)"
2

This was a Bug in consul 1.12.1 and is fixed in 1.12.2: `i/o deadline reached` using blocking queries on consul 1.12.1 · Issue #13303 · hashicorp/consul · GitHub