When using UI clients logs show: error="rpc error: Permission denied" rpc=ACL.GetPolicies

Nomad
1

The related clients show these errors when monitoring clients or jobs through UI. We are trying to understand where they come from.

nomad[1728258]:     2023-03-07T22:55:19.473Z [ERROR] client.rpc: error performing RPC to server: error="rpc error: Permission denied" rpc=ACL.GetPolicies server=172.16.8.36:4647
nomad[1728258]:     2023-03-07T22:55:19.473Z [ERROR] client.rpc: error performing RPC to server which is not safe to automatically retry: error="rpc error: Permission denied" rpc=ACL.GetPolicies server=172.16.8.36:4647
nomad[1728258]:     2023-03-07T22:55:19.473Z [WARN]  client: failed to resolve policies, using expired cached value: error="rpc error: Permission denied"

Here is more information regarding the agent we run:

nomad[1728258]: ==> Loaded configuration from /etc/nomad.d/nomad.hcl
nomad[1728258]: ==> Starting Nomad agent...
nomad[1728258]: 2023-03-07T22:54:43.582Z [TRACE] plugin.stdio: waiting for stdio data
nomad[1728258]: ==> Nomad agent configuration:
nomad[1728258]:        Advertise Addrs: HTTP: 172.16.8.37:4646
nomad[1728258]:             Bind Addrs: HTTP: [0.0.0.0:4646]
nomad[1728258]:                 Client: true
nomad[1728258]:              Log Level: INFO
nomad[1728258]:                 Region: global (DC: garage42)
nomad[1728258]:                 Server: false
nomad[1728258]:                Version: 1.5.0
nomad[1728258]: ==> Nomad agent started! Log data will stream in below:
2

Ok, so I was able to reproduce it by running the following:

curl ip:4646/v1/client/allocation/c29c50ff-af77-ed38-fca8-1db79ecc47a4/stats

But not by running the following:

curl -s -H "X-Nomad-Token: TOKEN" ip:4646/v1/client/allocation/c29c50ff-af77-ed38-fca8-1db79ecc47a4/stats

So then, I added the token to the UI, and the error disappeared. I guess I never realized UI worked even without the token.