What causes error "Operation on intention prefix denied due to ACLs'?

Nomad
1

Running nomad 0.11.0 with consul 1.7.2 with acls enabled.

On the consul servers, I see this error every few seconds

Apr 10 15:15:00 consul1 consul[28262]:     2020-04-10T15:15:00.797Z [WARN]  
agent.server.intentions: Operation on intention prefix denied due to ACLs: prefix=count-dashboard accessorID=12345678-1234-1234-1234-123456789012

On the nomad agents I see this error

Apr 10 15:25:41 nomadagent1 consul[6882]:     2020-04-10T15:25:41.564Z [ERROR] 
agent.client: RPC failed to server: method=Intention.Match server=10.47.69.130:8300 error="rpc error making call: Permission denied"
2

I was able to fix this by creating a policy for the nomad agents and applying it to /etc/nomad/config.json

I created a policy and token called consul-connect

agent_prefix "" {
    policy = "write"
}
node_prefix "" {
    policy = "write"
}
service_prefix "" {
    policy = "write"
}
acl = "write"

I then took the token and added it to the nomad config on the agents (not the servers)

/etc/nomad/config.json

{
  "consul": {
    "token": "123456"
  }
}

I then restarted the nomad and consul service on the agents

service nomad restart
service consul restart
4

Envoy

If you are using envoy and seeing this error, ensure that the envoy init script has CONSUL_HTTP_TOKEN as an environment variable

/etc/sysconfig/consul

CONSUL_HTTP_SSL=true
CONSUL_HTTP_ADDR=127.0.0.1:8501
CONSUL_CACERT=/etc/ssl/certs/foo.pem
CONSUL_CLIENT_CERT=/etc/foo.pem
CONSUL_CLIENT_KEY=/etc/foo.key
CONSUL_HTTP_TOKEN=12345678

Then inside the systemd unit file, ensure there is this line EnvironmentFile=-/etc/sysconfig/consul

[Unit]
Description=Start envoy proxy
Requires=local-fs.target
After=local-fs.target consul.service

[Service]
Type=simple
ExecStart=/usr/local/bin/consul connect envoy --sidecar-for foobar -admin-bind localhost:19000
EnvironmentFile=-/etc/sysconfig/consul

[Install]
WantedBy=multi-user.target
Wants=consul.service
5

Additional discussion https://github.com/hashicorp/consul/issues/7926