We are running a HashiCorp Boundary cluster in a high-availability (HA) configuration with the following high-level architecture:
- Controllers: 2 controller nodes running in active/active mode, distributed across different zones
- Workers: 1 worker per network, totaling more than 50 workers
Given the relatively heavy and extensive workload on this setup, I’m looking for guidance on how to effectively debug and troubleshoot the cluster.
So far, I’ve leveraged Prometheus metrics for observability, but they haven’t been very helpful in identifying root causes. Does Boundary provide any built-in tracing or advanced debugging capabilities that could offer deeper insights into its internal operations?
Additionally, I’ve noticed an issue with multi-hop scenarios: the metrics are only being reported for the ingress workers, while egress workers are not included in the metrics. This is problematic because our architecture maintains a relatively constant number of ingress workers, which makes the current metrics less useful for understanding overall worker health and performance.
I would greatly appreciate any recommendations or best practices on how to:
- Gain more visibility into the system’s internal behavior
- Diagnose and isolate worker-level issues (for example, determining if a specific worker is underperforming or misbehaving)
- Identify whether problems originate from the workers, controllers, or other parts of the system