Understanding Ingress gateways (particularly, Host -> Bridge gateways)

Hi there! I have the following job setup on some agents:

Agent 1:

  • Ingress gateway listening on hostnetwork:1234 and proxying to CoolService
  • CoolService on bridgenetwork:arbitraryportnumber

Agent 2:

  • Ingress gateway listening on hostnetwork:1234 and proxying to CoolService

Agent 3:

  • Ingress gateway listening on hostnetwork:1234 and proxying to CoolService

My expectation was that I’d be able to SSH into any of these agents and successfully curl localhost:1234; I cannot.
Instead, it only works on Agent 1 (where it’s co-located with the service in question).

So my questions are:

  1. Is my expectation - that the ingress gateway would work across agents - correct or no? (If my expectation is correct, then it sounds like there may be some broken Consul Connect shenanigans.)
  2. Does it even make sense to have the ingress gateway as a “system” job (i.e., running on every agent)?

You can find the definition of the ingress here:

You can find the definition of the service here (grapl-web-ui):

The motivation here is that I’d like to expose CoolService (well, grapl-web-ui) to the public internet behind an AWS API Gateway “Private Integration”.
You can learn more about that here: Set up API Gateway private integrations - Amazon API Gateway
The encapsulation roughly looks like:
request → API Gateway
→ Application Load Balancer
→ target group (basically, a group of EC2 instances healthily serving /v1/agent/health)
→ (okay, we are talking to a Nomad agent box right now; we don’t know if it has CoolService on it or not)
→ Nomad Ingress gateway serving at :1234
→ Nomad web-ui