Consul acl policy not working as expected

Consul
1

I’ve enabled Consul ACL and having some “not-so-nice” error messages in the logfile of consul:

Aug 01 16:20:04 patroni001-twolf-test consul[2565]:     2019/08/01 16:20:04 [ERR] http: Request GET /v1/kv/service/pg-cluster/leader?consistent=&index=403382&wait=1000ms, error: Permission denied from=127.0.0.1:40904

The policy i’m using is this one:

{
  "key_prefix": {
    "vault/": {
      "policy": "write"
    }
  },
  "key_prefix": {
    "service/": {
      "policy": "write"
    }
  },
  "node_prefix": {
    "": {
      "policy": "write"
    }
  },
  "service": {
    "vault": {
      "policy": "write"
    }
  },
  "agent_prefix": {
    "": {
      "policy": "write"
   }

  },
  "session_prefix": {
    "": {
      "policy": "write"
    }
  }
}

Am i missing something, because everything under the key “service” should be available to the agent?

2

@Wolfsrudel How have you setup the policy. I am assuming you are using the new APIs/CLI and not attaching the policy directly to a legacy ACL token. The only reason I ask is that the *_prefix ACL resources would not be recognized and could cause what you are seeing.

A second cause would be if your token is linked with multiple policies and one of those policies explicitly denies access to that part of the KV store.

Nothing else comes to mind why this would happen but if you can provide a little more information about how the token and policy were created then it might help.

3

@mkeeler Thanks for your help. After bootstrapping consul acl the token is the first - and only - i’ve created so far.

consul acl bootstrap
CONSUL_HTTP_TOKEN=<token>
export CONSUL_HTTP_TOKEN
consul acl policy create \
  -name "patroni" \
  -datacenter "${DATACENTER}" \
  -rules @consul.policy.patroni.acl
consul acl token create \
  -description "patroni agent token" \
  -policy-name "patroni"

This token is used in the configuration of patroni. Nevertheless everything is working as expected.
In addition, this annoying error message appears.

4

So I just tried:

  1. consul agent -dev -hcl 'acl {enabled = true default_policy = "deny"}'
  2. consul acl bootstrap
  3. consul acl policy create -name patroni -datacenter dc1 -rules @policy.json (I copied your policy into policy.json and am using the bootstrap token)
  4. `consul acl token create -description “test” -policy-name “patroni” (again using the bootstrap token)
  5. consul kv put -token <new token secret> service/pg-cluster/leader 1
  6. consul kv get -token <new token secret> service/pg-cluster/leader

Neither of those final two are giving back a permission denied error.

I suspect that something is making a request without specifying a token. It is then using the anonymous token which exists but doesn’t have the required privileges.

You could test out this assumption by running:

consul acl set-agent-token default <your new token id>

If the messages go away then whatever was making the requests was doing so without a token being set. Note that you could also link the policy to the anonymous token but that affects the whole cluster instead of just the one agent.

Afterwards you probably want to unset that default token with:

consul acl set-agent-token default ""

If you need that default access for the agent and are only serving the HTTP API on the loopback interface (this is the default and is overridden with -client <addr> CLI parameter) then you can configure the default token within the configuration like so.

acl {
   tokens {
      default = <token secret id linked to your policies>
   }
}

Or

acl {
   enable_token_persistence = true
}

followed by the consul acl set-agent-token default <token secret>.

Normally tokens set via the API/CLI will not persist between restarts of Consul but by setting that config value it enables that persistence to work.

1 Like
5

Thanks a lot, I’ll give it a try tomorrow.

6

Nice, it’s gone now. The rest is up to me. Thank you very much!

7

Perfect, just so I am not leading you down a route you don’t want to go I will say that setting the default token allows anything that can reach the HTTP API to use that token. That very well could be ok for you and meet your security requirements but its something to be aware of.

Keeping the client address set to 127.0.0.1 does minimize what can talk to Consul and gain usage of that token. I just wanted to make sure I wasn’t leading you into something less secure.