Consul ACL not letting sidecar-proxies been registered

Consul
1

I have Consul ACL enabled and I am using the following policy

    node "" { 
      policy = "write"
    } 
    agent "" { 
      policy = "write"
    } 
    event "" { 
      policy = "write"
    } 
    key "" { 
      policy = "write"
    } 
    query "" { 
      policy = "write"
    } 
    service "" { 
      policy     = "write"
      intentions = "write"
    } 
    session "" { 
      policy = "write"
    }

I set the corresponding token everything works as expected only the sidecar-proxies are not.
This the following output that I am having:

    2019/10/19 19:08:24 [DEBUG] agent: Check "_nomad-check-6bd37b130919c3381a50c3baecac15dd748b4815" in sync
    2019/10/19 19:08:24 [DEBUG] agent: Check "_nomad-check-e54e94f2ff76b872641b9ca95d41d7f9c864c22d" in sync
    2019/10/19 19:08:24 [DEBUG] agent: Check "_nomad-check-ec6c26a2c5515a96ed9406eb522a13addfc46fc5" in sync
    2019/10/19 19:08:24 [DEBUG] agent: Check "service:_nomad-task-4b9d2d70-08f7-30ae-10cd-ab819c8fd074-group-payment-api-payment-api-3000-sidecar-proxy:2" in sync
    2019/10/19 19:08:24 [DEBUG] agent: Check "service:_nomad-task-1e0263d9-1119-f0c2-c728-afa999ea115b-group-notification-api-notification-api-3001-sidecar-proxy:1" in sync
    2019/10/19 19:08:24 [DEBUG] agent: Check "service:_nomad-task-1e0263d9-1119-f0c2-c728-afa999ea115b-group-notification-api-notification-api-3001-sidecar-proxy:2" in sync

    2019/10/19 19:13:39 [DEBUG] Error handling ADS stream: rpc error: code = PermissionDenied desc = permission denied
    2019/10/19 19:13:39 [DEBUG] http: Request GET /v1/agent/services (250.559µs) from=172.20.20.11:50982
    2019/10/19 19:13:39 [DEBUG] http: Request GET /v1/agent/checks (197.593µs) from=172.20.20.11:50982
    2019/10/19 19:13:41 [DEBUG] agent: Check "_nomad-check-79ade9b3bc4d0909064d3dbac6ac33874d1cdf2b" is passing
    2019/10/19 19:13:41 [DEBUG] agent: Check "_nomad-check-ec6c26a2c5515a96ed9406eb522a13addfc46fc5" is passing
    2019/10/19 19:13:41 [DEBUG] agent: Check "_nomad-check-6bd37b130919c3381a50c3baecac15dd748b4815" is passing
    2019/10/19 19:13:42 [WARN] agent: Check "service:_nomad-task-390723c8-7fd4-f0fa-0068-560526b36135-group-payment-api-payment-api-3000-sidecar-proxy:1" socket connection failed: dial tcp 127.0.0.1:26581: connect: connection refused
    2019/10/19 19:13:42 [DEBUG] Error handling ADS stream: rpc error: code = PermissionDenied desc = permission denied

Even when I change my consul token to my global-management token, I am having the same output.

my services are been registered using nomad, and the sidecar proxies as well with the latest nomad beta version.

this what I get in the UI

24%20PM
Screen Shot 2019-10-19 at 2.16.24 PM.png1920×995 104 KB

2

Does this issue is related on how envoy is authenticated with ACLs enabled ?

is it related to this function https://github.com/hashicorp/consul/blob/973341a5926679c548ff99809eb6d6ee62c2c1c6/agent/xds/server.go#L150

3

I have found a worked around to this, at initial bootstrap, I need to have acl in allow mode, then setup all acl configurations, then change to deny, and re generate the token, after this and with proper acl policies, sidecar services are registered without issue.

4

I tried that, but it did not work for me.