When using UI clients logs show: error="rpc error: Permission denied" rpc=ACL.GetPolicies

Nomad
1

The related clients show these errors when monitoring clients or jobs through UI. We are trying to understand where they come from.

nomad[1728258]:     2023-03-07T22:55:19.473Z [ERROR] client.rpc: error performing RPC to server: error="rpc error: Permission denied" rpc=ACL.GetPolicies server=172.16.8.36:4647
nomad[1728258]:     2023-03-07T22:55:19.473Z [ERROR] client.rpc: error performing RPC to server which is not safe to automatically retry: error="rpc error: Permission denied" rpc=ACL.GetPolicies server=172.16.8.36:4647
nomad[1728258]:     2023-03-07T22:55:19.473Z [WARN]  client: failed to resolve policies, using expired cached value: error="rpc error: Permission denied"

Here is more information regarding the agent we run:

nomad[1728258]: ==> Loaded configuration from /etc/nomad.d/nomad.hcl
nomad[1728258]: ==> Starting Nomad agent...
nomad[1728258]: 2023-03-07T22:54:43.582Z [TRACE] plugin.stdio: waiting for stdio data
nomad[1728258]: ==> Nomad agent configuration:
nomad[1728258]:        Advertise Addrs: HTTP: 172.16.8.37:4646
nomad[1728258]:             Bind Addrs: HTTP: [0.0.0.0:4646]
nomad[1728258]:                 Client: true
nomad[1728258]:              Log Level: INFO
nomad[1728258]:                 Region: global (DC: garage42)
nomad[1728258]:                 Server: false
nomad[1728258]:                Version: 1.5.0
nomad[1728258]: ==> Nomad agent started! Log data will stream in below:
2

Ok, so I was able to reproduce it by running the following:

curl ip:4646/v1/client/allocation/c29c50ff-af77-ed38-fca8-1db79ecc47a4/stats

But not by running the following:

curl -s -H "X-Nomad-Token: TOKEN" ip:4646/v1/client/allocation/c29c50ff-af77-ed38-fca8-1db79ecc47a4/stats

So then, I added the token to the UI, and the error disappeared. I guess I never realized UI worked even without the token.

3

It looks like a permission issue in the Nomad agent, causing errors in RPC calls to the server. Check the ACL policies and ensure proper permissions for monitoring clients/jobs. The agent’s configuration DIY Garage Door seems fine, but there might be a misconfiguration in ACL or a server permission issue. Reviewing and updating the ACL policies should help resolve this.

4

It seems like there’s a permission issue with Nomad RPC when monitoring clients or jobs through the UI. The errors indicate “rpc error: Permission denied.” Have you checked Garage Stops ACL policies on the server at 172.16.8.36:4647? It might be worth reviewing and adjusting the permissions in your Nomad configuration to resolve these access problems. :hammer_and_wrench::mag:

5

It seems like there’s a permission issue with Nomad, causing errors in RPC communication with the server. Check the ACL policies on the server at 172.16.8.36:4647. Ensure that the Nomad Troubleshoot agent at 172.16.8.37:4646 has the necessary permissions. Review the Nomad agent configuration to confirm the settings match your requirements.