Terminating gateway broken with workload identities

Nomad
1

I’m migrating a cluster to use workload identities for both vault and consul. While I think I get most things working, my terminating gateway is broken. As soon as I start it with workload identities for consul enabled, the consul agent logs of the node running the gateway is full of errors like this

oct. 01 11:17:14 ct-w-out consul[4368]: 2024-10-01T11:17:14.719+0200 [WARN]  agent.cache: handling error in Cache.Notify: cache-type=trust-bundles error="rpc error: code = Unknown desc = Permission denied: token with AccessorID 'fb1a7436-3261-3b82-f97f-256f692f1e4a' lacks permission 'service:write' on \"ldap\"" index=0
oct. 01 11:17:14 ct-w-out consul[4368]: 2024-10-01T11:17:14.719+0200 [ERROR] agent.proxycfg: Failed to handle update from watch: kind=terminating-gateway proxy=_nomad-task-a22c5142-9094-c6a0-1284-14c6eed352db-group-egress-gateway-egress-gateway-connect-terminating-egress-gateway service_id=_nomad-task-a22c5142-9094-c6a0-1284-14c6eed352db-group-egress-gateway-egress-gateway-connect-terminating-egress-gateway id=peer-trust-bundle:ldap error="error filling agent cache: rpc error: code = Unknown desc = Permission denied: token with AccessorID 'fb1a7436-3261-3b82-f97f-256f692f1e4a' lacks permission 'service:write' on \"ldap\""

ldap is one of the external service exposed in the mesh by my terminating gateway. The service itself is registered in consul. The issue seems to be that the default ACL doesn’t allow write access to this service. Anyone has some info on the required ACL/binding rules needed for terminating gateways to work ?

2

I guess I need to create a dedicated consul policy, and a role. But I fail to understand how to create the binding rule so my terminating gateway has this new role instead of the default one

3

I think I finally got it. For anyone else wondering, first, create a cutom consul policy, granting write access to the service you intend to expose with the terminating gateway

key_prefix "" {
  policy = "read"
}

node_prefix "" {
  policy = "read"
}

service_prefix "" {
  policy = "read"
}
service "db-mysql" {
  policy = "write"
}
service "imap" {
  policy = "write"
}
service "ldap" {
  policy = "write"
}
service "proxyout" {
  policy = "write"
}

(same as the default policy, just adds write for my exposed services, in this example db-mysq, ldap, imap, proxyout)

Then create a binding rule like this

consul acl binding-rule create -method=nomad-workloads \
  -bind-type=policy \
  -bind-name=egress-gateway \
  -selector="value.nomad_job_id==\"egress-gateway\""

In my case, the terminating gateway is in a job named egress-getway

4

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.