Requesting help to resolve log spamming with /v1/agent/self error="Permission denied" errors

Consul
#1

Hi All:

Requesting help figuring out why my Consul agents are spamming my logs with: [ERROR] agent.http: Request error: method=GET url=/v1/agent/self from=127.0.0.1:60948 error=“Permission denied”.

This is using Consul 1.8.5 and all agents are running in Docker containers. ACL’s have been enabled and I’ve set up agent policies and tokens per the Secure Consul documentation. As far as I can tell everything is syncing

Here’s the policy:
ID: 6277eb3c-3152-4188-a3e5-3bda65f6b032 Name: agent_stage-appserver01 Description: Consul Agent Policy for: stage-appserver01 Datacenters: stage01 Rules: node “stage-appserver01” { policy = “write”}

and here is the token:
AccessorID: 2f03db40-44c9-b48d-8202-c710a37a6b13 Description: Consul Agent Token for stage-appserver01 Local: false Create Time: 2020-11-06 21:01:32.352564583 +0000 UTC Legacy: false Policies: 6277eb3c-3152-4188-a3e5-3bda65f6b032 - agent_stage-appserver01

The requests generating this messages are coming from 127.0.0.1 and a 60948 or 60954 port.

Any help would be much appreciated.

#2

Finally found the problem - a nomad instance that I thought was disabled.

FWIW, I found this very difficult to track down given the lack of information in the log (even at TRACE level). It would have been helpful had some info on the source of the message been available. I would think this would be a nightmare to track down given tens of services.

-steve