Confusion in HCP Boundary Docs: Should Workers Connect to Port 9201 of Controllers Directly or via Load Balancer?

Boundary
1

Hi everyone,

I noticed a potential inconsistency in the HCP Boundary documentation regarding how ingress workers should connect to controller port 9201.

On this page, there’s a diagram that shows ingress workers connecting directly to port 9201 on the controllers. However, in another section, the configuration example suggests connecting to port 9201 through a load balancer instead. ( Line 14 of /etc/boundary.d/ingress-worker.hcl suggests using ‍‍<contoller_lb_address>:9201)

This raised a question:
Which approach is actually recommended for production deployments of HCP Boundary?

I understand that the controller API (9201) handles operational traffic like authentication and session brokering, but I’d like to confirm whether direct access to controllers is ever appropriate — or if all connections should be routed through a load balancer for reliability and scaling.

Another thing that I’ve couldn’t wrap my head around is that are controllers need to communicate with each other? because in Vault we see that the nodes have identity and we cannot use loadbalancer in front of the port 8002 ( I guess the port is right ), but my assumption here is that the controllers are just stateless and they do not know how many of them are deployed. In other words is this sentence correct ?
The Boundary controllers have 0 dependency to each other, that means the cryptographic operations are happening in the vault ( or through a shared key ) so that everything that one node signs or trusts the others can verify as if it was signed or trusted by themselves.

2

Hi @mehrdad-khojastefar , thanks for pointing out this inconsistency, I’ll raise it with our docs team.

We do recommend using a load balancer for prod deployments: System requirements | Boundary | HashiCorp Developer

That is correct, controllers have no dependency on each other. State is maintained in the database and only controllers have access to the database.
“The Boundary controllers do not directly communicate with one another, all configuration and state is managed through an RDBMS, in this case PostgreSQL.” from:

3

Thanks for your answer.
I see that communication on port 9201 is not http. What protocol is being used here ?

4

Maybe useful for future readers.

The protocol is being used I think is gRPC.
Boundary controller and workers communicate over TLS using ALPN.

      frontend boundary-9201
        bind 0.0.0.0:9201
        mode tcp

        use_backend boundary-9201

      backend boundary-9201
        mode tcp

        option tcp-check
        tcp-check connect

        server n1-boundary n1.boundary.x.x:9201 check
        server n2-boundary n2.boundary.x.x:9201 check

This HAProxy config did the job.