Log Destination for Agent Watches

Consul
1

I’m experimenting with adding “executable” watch handlers to my consul agents to respond to K/V updates. I noticed on the documentation page that it states: “Anything written to stdout is logged”. Unfortunately, I’m having a hard time finding this logged output. My consul agents are running as systemd service units, with the following unit file:

[Unit]
Description="HashiCorp Consul - A service mesh solution"
Documentation=https://www.consul.io/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/consul.d/consul.hcl

[Service]
Type=notify
User=consul
Group=consul
ExecStart=/usr/bin/consul agent -config-dir=/etc/consul.d/
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGTERM
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

I only have one file in the config-dir, which contains:

datacenter = "us-east-2-5-0-0-snapshot-proto"
data_dir = "/opt/consul"
node_name = "mynode-clean-xyz"
retry_join = [ "10.0.80.178" ]
encrypt = "..."
watches = [
    {
        type = "keyprefix"
        prefix = "mycompany/tenant/clean"
        handler_type = "script"
        args = [ "/tmp/handle.sh" ]
    }
]

When I update a value matching my keyprefix, my handler at /tmp/handle.sh is definitely invoked; I can see its effects with each K/V update. However, I can’t locate the messages it’s echoing to STDOUT. I’ve tried:

tail -f /var/log/daemon.log

and

journalctl -fu consul.service

Both locations show the [INFO] messages from consul bootup, but no subsequent messages from my watch handler invocations.

2

Figured it out. Adding log_level = "DEBUG" to the configuration caused the text in my handler scripts’ STDOUT to show up in the journalctl logs. Now I’m seeing entries like this:

2021-02-17T14:55:01.621Z [DEBUG] agent: watch handler output: watch_handler=[/tmp/handle.sh]
<STDOUT from /tmp/handle.sh>