Hello, I am installing a cluster with nomad and consul, I have managed to deploy jobs in the normal way but when I try to add the sidecar_service block I get an error that says: No ACL token was provided to Envoy. Because the ACL system is enabled, pass a suitable ACL token for this service to Envoy to avoid potential communication failure.
failed fetch proxy config from local agent: Unexpected response code 403: (Permission denied: anonymous token lacks permission ‘service:read’ on ‘jobname-sidecar-proxy’). The anonymous token is used implicitly when a request does not specify a token.)
I need your help please
1 Like
Hi @aoviedosolis ,
When Consul is ACL enabled, you must configure Nomad with Workload Identity to generate on-demand ACL tokens for Services that uses Consul.
This documentation will help you set up the same.
ref: Configure Consul ACL with Nomad Workload Identities | Nomad | HashiCorp Developer
1 Like
Should I configure and setup this before enabling ACL and TLS or not? cause even in documentation it use http://127.0.0.1:4646 instead of IP or domain of that service with TLS and https for ex: https://10.10.40.3:4646 I even tried to check and set this up with following environment variables but I got error
sudo su - root
export CONSUL_HTTP_ADDR=https://10.10.40.3:8501
export NOMAD_ADDR=https://10.10.40.3:4646
export CONSUL_HTTP_TOKEN="xyz-my-bootstrap-token"
export NOMAD_TOKEN="xyz-my-bootstrap-token"
export CONSUL_CACERT=/etc/consul.d/consul-ca.pem
export CONSUL_CLIENT_CERT=/etc/consul.d/cli.pem
export CONSUL_CLIENT_KEY=/etc/consul.d/cli-key.pem
export NOMAD_CACERT=/etc/nomad.d/nomad-ca.pem
export NOMAD_CLIENT_CERT=/etc/nomad.d/cli.pem
export NOMAD_CLIENT_KEY=/etc/nomad.d/cli-key.pem
nomad setup consul -jwks-url=https://10.10.40.3:4646/.well-known/jwks.json -jwks-ca-file=/etc/nomad.d/nomad-ca.pem
They are both on the same server, the firewall is disabled and I even export CONSUL_HTTP_SSL_VERIFY=false but didn’t work 
error: Nomad JWKS endpoint unreachable, verify that Nomad is running and that the JWKS URL https://10.10.40.3:4646/.well-known/jwks.json is reachable by Consul
There is no youtube video or blog to guide and walk through
Hi @mehdiMj-ir ,
You should only be running the nomad setup consul after you have ACL enabled on both Nomad and Consul.
While the documentation shows a non-tls example, it is important to run Nomad and Consul with TLS in production environments.
I would recommend checking whether nomad is actually binding to that IP or not.
What is the output of netstat -antlp | grep 4646?
You could also try replacing the ip of the -jwks-url argument with https://127.0.0.1:4646/.well-known/jwks.json
I hope this helps.
1 Like
Thank you for your response. Yes, both Nomad and Consul ACLs and TLS were enabled.
One important point that should be mentioned in the documentation is the need to disable verify_https_client, because if client TLS verification is a hard requirement despite ACLs being enabled.
I resolve the issue from this quote and finally able to update from Nomad 1.9.7 to 1.11.1 on production! (this is me test environment)
Here is that quote
ss -nrpl # for more infoI run Nomad and Consul behind an Nginx reverse proxy on port 2096, with Cloudflare serving them over HTTPS on port 443.
1 Like