Consul-connect job results in "ACL support disabled" error

Nomad
1

job spec.hcl (722 Bytes)
consul_server_config.json (1.2 KB)
nomad_server_config.hcl (1.0 KB)
Nomad version: v0.12.1
Consul version: v1.7.2

We have a setup of 3 nomad servers, 3 nomad clients and 3 consul servers; each on their own VM. Everything seems to be working properly together. However recently we have tried to add a new service to our nomad cluster that utilizes consul-connect. Everything works great locally using -dev and -dev-connect options. However recently we have tried to add the first service to our nomad cluster that utilizes consul-connect. (To be clear we are not using -dev flags in our testing env)

Aug 18, 20 11:16:44 -0600	Alloc Unhealthy	Unhealthy because of failed task
Aug 18, 20 11:16:44 -0600	Not Restarting	Error was unrecoverable
Aug 18, 20 11:16:44 -0600	Task hook failed	consul_si_token: Unexpected response code: 401 (ACL support disabled)
Aug 18, 20 11:16:44 -0600	Killing	failed to derive SI token: Unexpected response code: 401 (ACL support disabled)

We have tried to open policy’s to everything. default_policy: allow and a bunch of other random things to no avail.

Any help would be extremely welcomed.

Thanks for reading friend!

2

Did you bootstrap the acl system?

consul acl bootstrap

Just enabling it using the config file isn’t enough. :wink:

3

Hi @jweispfenning-jc,

I see that you have ACLs enabled in your Consul configuration, but the error message returned by Nomad suggests that they may not be fully bootstsrapped.

If you haven’t already, I recommend checking out the guide Secure Nomad Jobs with Consul Service Mesh for a walkthrough of bootstrapping Consul ACLs, and configuring Nomad to work with Connect.

If you’ve already bootstrapped the ACL system and are still running into issues, then the next step may be to verify the ACL policies are correctly configured to allow Nomad to create the service tokens in Consul.

4

Alright looks like we found the issue… and as I thought it was dumb (on our part). We enabled acl on the consul servers but not on the consul agents. enabled it there and worked perfectly. Thanks for the responses!