Not sure if this is intended behaviour so I am checking in with the community.
We use consul-template to listen for KV store changes and render the Nomad HCL job definitions after which command is executed to submit new job definition to Nomad cluster.
This is all good and working. Now what we are might be abusing is a fact that any of the templates that are missing KV values will not be rendered thus not be submitted to Nomad cluster. This was a quick and easy way for us to decide which templates not to deploy in a particular environment and still keep all the templates in the same git repo.
And here is the “issue” we are noticing when keeping “non rederable templates” because of missing KV values.
We are using the wait parameter so we don’t have many renders happen if multiple KV store values are changed. But in the case when KV params are missing for some of the templates we get multiple renders happen even though we provided wait parameter.
consul-template never gets to quiescence stage. Going through parameters in the docs I could not find an obvious option to “skip” the problematic template.
In logs we should see similar to this example:
2023-09-14T07:29:10.379Z [DEBUG] (runner) watching 4 dependencies
2023-09-14T07:29:10.380Z [DEBUG] (runner) all templates rendered
2023-09-14T07:29:10.380Z [DEBUG] (runner) enabling global quiescence for "1d56665649284454fd46ad5c8899f3b8"
2023-09-14T07:29:10.381Z [DEBUG] (runner) enabling global quiescence for "28bf72edc653729a4304301c82492b2d"
Instead we have consul-log like this (last line) and never get to quiescence state:
2023-09-14T07:30:49.758Z [DEBUG] (runner) watching 4 dependencies
And when we update KV store our “valid” templates do get rendered but we get multiple triggers to render the same template and multiple commands are fired.
Again easy fix was just to remove these templates and all is fine but we are wondering if this is possibile “edge case” and maybe a bug.
If anyone is interested to test this out here are quick steps to test working and non working examples.
Working Example (I use two terminal windows for easier testing and template output check)
mkdir /tmp/consul-template-double-deploy/ && cd /tmp/consul-template-double-deploy/
export DOCKER_NETWORK_NAME=consul-template-quiescence-issue
docker network create $DOCKER_NETWORK_NAME
# Run consul and sleep two seconds
docker run -d --rm --network-alias consul-kv --network $DOCKER_NETWORK_NAME --name consul-kv hashicorp/consul:1.15.4 && sleep 2
# Prepopulate kv
docker exec consul-kv consul kv put first_value 100 \
&& docker exec consul-kv consul kv put second_value 200 \
&& docker exec consul-kv consul kv put third_value 300 \
&& docker exec consul-kv consul kv put fourth_value 400
# Create WORKING templates example files
cat <<EOF > tpl_1.ctmpl
{{ key "first_value" }}
{{ key "second_value" }}
EOF
cat <<EOF > tpl_2.ctmpl
{{ key "third_value" }}
{{ key "fourth_value" }}
EOF
# Run WORKING consul-template example
docker run --rm \
-e "CONSUL_HTTP_ADDR=consul-kv:8500" \
-v $(pwd):/tpl \
--network $DOCKER_NETWORK_NAME \
--name consul-tpl \
hashicorp/consul-template:0.33 \
-log-level=debug \
--wait=3s \
-template '/tpl/tpl_1.ctmpl:/tpl/tpl_1.rendered:echo "$(date)" >> /tpl/tpl_1.deployed' \
-template '/tpl/tpl_2.ctmpl:/tpl/tpl_2.rendered:echo "$(date)" >> /tpl/tpl_2.deployed'
Then update first_value and second_value you should se only one new line added to the *.deployed file. Eg.
docker exec consul-kv consul kv put first_value 440 & docker exec consul-kv consul kv put second_value 330
CLEANUP
docker kill consul-kv
docker kill consul-tpl
docker network rm consul-template-quiescence-issue
rm -f tpl*
If you want to check non working example, delete all files (cleanup), repeat same commands from above but don’t add fourth_value to KV store.
First template will be rendered. Second template will never get rendered.
Then update first_value and second_value to get command executed twice and you should see two new lines added to tpl_1.deployed
I hope this makes sense and is easy to reproduce.