Newbie: Service Registration Blocked by ACL

marquis.of.sandwiche · October 12, 2023, 4:35am

Hi again,
I have a consul cluster running as the orchestrator for Postgresql. (Patroni)
I followed these instructions on one consul instance and it worked as described. Secure Consul with access control lists (ACLs) | Consul | HashiCorp Developer

Adding a second consul node gets this on the first consul node:

[ERROR] agent.http: Request error: method=PUT url=/v1/agent/service/register from=192.168.2.5:42336 error="Permission denied: token with AccessorID '2696e7d3-cb6a-6004-55fe-0c514ba1b7e3' lacks permission 'service:write' on \"postgres\""
2023-10-11T21:32:33.606-0700 [WARN]  agent: Service registration blocked by ACLs: service=47e7ed89-5c11-b681-29e4-6ef6b2c6d490 accessorID=47e7ed89-5c11-b681-29e4-6ef6b2c6d490

How do I construct an acl to allow .5 to register on .7?

Do I reapply the same acl I used on .7 on .5? (using .5’s token???)

Ranjandas · October 12, 2023, 9:16am

Hi @marquis.of.sandwiche,

Are you not using service-identity tokens as described in this link: Secure Consul with access control lists (ACLs) | Consul | HashiCorp Developer

From the error, it looks like your tokens policy doesn’t allow write for the service named postgres.

What do you mean by .5 and .7 in this context? If they are different VM instances, yes, you can use the same token on both servers.

But from a security best practice point of view, it is better you repeat that service-identity command and create different tokens for each server. It all depends on your preference.

marquis.of.sandwiche · October 15, 2023, 2:49am

Thanks for the reply. .5 and .7 are the suffixes to a couple of VMs. I deleted everything in /opt/consul and started from scratch for the following:
config.hcl has

acl {
  enabled = true
  default_policy = "allow"
  enable_token_persistence = true
}

created bootstrap token.
create agent token on each host. (.5 and .7 VMs)
create service token. (I assume this is global???)
I confirmed they are there using the global token
export CONSUL_HTTP_TOKEN="global-uuid" ; consul acl token list
Switch to agent’s token.
export CONSUL_HTTP_TOKEN=".5-uuid" ; consul acl token list
As expected, access is denied.
export CONSUL_HTTP_TOKEN=“.7-token” ; consul acl policy create -name “dot-seven-service” -description “allow .7 service” -rules @consul-acl-policy.hcl
Access denied which, makes sense. Then using the global uuid, it works as expected.
This is what’s in consul-acl-policy.hcl

service_prefix "" {
   policy = "write"
}

session_prefix "" {
  policy = "write"
}

key_prefix "" {
  policy = "write"
}

node_prefix "" {
  policy = "read"
}

agent_prefix "" {
  policy = "read"
}


service "postgres" {
 policy = "write"
}

Reload each node just in case there’s something to be had doing so.
Start patroni (it’s running/logging to stdout on another shell.)
Error:
[ERROR] agent.http: Request error: method=PUT url=/v1/agent/service/register from=192.168.2.5:52268 error=“Permission denied: token with AccessorID ‘9f43f7d6-ac59-fadf-9975-4f3ece8096be’ lacks permission ‘service:write’ on "postgres"”

uuid ending in be is .7

.7 still doesn’t have access despite the acl applied with the global secret.

Not sure where to go from here.

marquis.of.sandwiche · October 15, 2023, 10:36pm

There’s no obvious directions in the documentation once you apply the ACL.

The next step is to create a role that marries the ACL policy(ies) , service identities, and node-identities.
CONSUL_HTTP_TOKEN=“global-key” consul acl role create -name “register” -description “service register role” -policy-name “bar” -policy-name “roo” -service-identity “postgres” -datacenter “batcave” -node-identity “zebra:batcave” -node-identity “giraffe:batcave” -node-identity “okapi:batcave”

But, this still doesn’t permit the service from registering. My gut feeling is, something about the URL needs an acl. But, no idea what.

[ERROR] agent.http: Request error: method=PUT url=/v1/agent/service/register from=192.168.2.5:59038 error="Permission denied: token with AccessorID ‘.5-uuid’ la
cks permission ‘service:write’ on "postgres""

Ranjandas · October 15, 2023, 11:11pm

Hi @marquis.of.sandwiche,

Could you share the output of the following command?

consul acl token read -id <.5-uuid> -token <privileged-token> -expanded

The errors says that the token you are using to register the postgres service doesn’t have the required permissions.

marquis.of.sandwiche · October 16, 2023, 12:10am

Hi, thanks so much for your help.

AccessorID:       postgres-uuid
SecretID:        foo
Description:      Token for postgres
Local:            false
Create Time:      2023-10-15 14:42:57.220490726 -0700 PDT
Service Identities:
        Name: postgresql (Datacenters: all)
                Description: synthetic policy for service identity "postgresql"
                Rules:
                        service "postgresql" {
                                policy = "write"
                        }
                        service "postgresql-sidecar-proxy" {
                                policy = "write"
                        }
                        service_prefix "" {
                                policy = "read"
                        }
                        node_prefix "" {
                                policy = "read"
                        }

=== End of Authorizer Layer 0: Token ===
=== Start of Authorizer Layer 2: Agent Configuration Defaults (Inherited) ===
Description: Defined at request-time by the agent that resolves the ACL token; other agents may have different configuration defaults
Resolved By Agent: ".7-host"

Default Policy: allow
        Description: Backstop rule used if no preceding layer has a matching rule (refer to default_policy option in agent configuration)

Down Policy: extend-cache
        Description: Defines what to do if this Token's information cannot be read from the primary_datacenter (refer to down_policy option in agent configuration)

And then, here’s the same info for the .7 token

AccessorID:       .7-token-id
SecretID:         bar
Description:      Token for zebra
Local:            false
Create Time:      2023-10-15 14:45:59.339792967 -0700 PDT
Node Identities:
        Name: zebra (Datacenter: batcave)
                Description: synthetic policy for node identity "zebra"
                Rules:
                        node "zebra" {
                                policy = "write"
                        }
                        service_prefix "" {
                                policy = "read"
                        }

=== End of Authorizer Layer 0: Token ===
=== Start of Authorizer Layer 2: Agent Configuration Defaults (Inherited) ===
Description: Defined at request-time by the agent that resolves the ACL token; other agents may have different configuration defaults
Resolved By Agent: "zebra"

Default Policy: allow
        Description: Backstop rule used if no preceding layer has a matching rule (refer to default_policy option in agent configuration)

Down Policy: extend-cache
        Description: Defines what to do if this Token's information cannot be read from the primary_datacenter (refer to down_policy option in agent configuration)

Ranjandas · October 16, 2023, 1:09am

Hi @marquis.of.sandwiche,

Thank you for sharing the output. If you compare the error log and the policy, you will see that, you are trying to register the service with the name postgres, but your policy only allows for the service named postgresql.

With the token you shared, you will be able to register the service with the name postgresql. So to fix the issue, either change the service name to postgresql, or create and use a token with service-identity for service named postgres.

I hope this helps.

marquis.of.sandwiche · October 16, 2023, 2:39am

I now have a service token for postgresql. Same results.
[ERROR] agent.http: Request error: method=PUT url=/v1/agent/service/register from=192.168.2.176:37474 error=“Permission denied: token with AccessorID ‘099c6009-ceb1-94b7-7cfb-6a5efcee43e5’ lacks permission ‘service:write’ on "postgres"”

Patroni still, apparently is trying to use the name postgres and is denied. No amount of permissive settings seems to work.

Ranjandas · October 16, 2023, 2:51am

Can you share the output of:

consul acl token read -id 099c6009-ceb1-94b7-7cfb-6a5efcee43e5 -token <privileged-token> -expanded

Please only mask the secretid so that I can see the output is from the token with the correct accessor-id.

You should have a service-identity token for postgres, as whoever is trying to register the service is using the name postgres.

In addition, I think your local agent has a default_policy of “deny”, which is why it is denying the request. I would say keep the default_policy as deny, and get the policy fixed.

marquis.of.sandwiche · October 16, 2023, 3:18am

Thank you for your continued patience with this.

You got me moving in the right direction. For reasons I have yet to sort out, despite granting postgres the exact same rights as postgresql, I changed the configuration on patroni to use the service name postgresql, and it worked.

I still get this error, but overall it actually seems to work right.
agent.http: Request error: method=GET url=/v1/agent/service/deregister/postgres/police-van from=192.168.2.176:34470 error=“method GET not allowed”
I am guessing that an acl like agent “http” { policy = “write” } fixes that??

Ranjandas · October 16, 2023, 3:38am

Hi @marquis.of.sandwiche,

I am glad that you got it working.

This error has nothing to do with the ACL permissions. It just says that HTTP Method GET is not allowed. To deregister a service it should be HTTP PUT.

ref:

I hope this helps.

Topic		Replies	Views
Consul acl policy not working as expected Consul acl	6	3045	August 6, 2019
Really struggling with ACLs Consul	4	326	September 7, 2021
Confused over applying basic ACLs Consul	1	434	November 8, 2019
HCSEC-2021-24 - Consul Missing Authorization Check on Txn.Apply Endpoint Security security-consul	0	7715	September 1, 2021
Consul exclusive ACL policy for nodes/service in ASG Consul	4	774	July 12, 2019

Newbie: Service Registration Blocked by ACL

Related topics