Here is a trace of the first 60 seconds while Boundary 0.5.1 in dev mode starts up and initializes its database trace.out.txt (1.4 MB). Eventually, rename it to trace.out, hopefully it does not get mangled by up- & downloads which limits file extensions).
You should be able to run go tool trace -http=127.0.0.1:6060 trace.out on any platform, and then view it in (exclusively!) Chrome at http://127.0.0.1:6060.
I have not inspected it in detail yet, but want to generate also the various outputs that pprof is able to generate.
Them combined will hopefully provide ideas where to dig next. Eventually, Boundary just runs out of resources. Although I had considerably raised the limit of openfiles for example already, which is also necessary to run Consul and Vault on OpenBSD, otherwise they misbehave too.
Here are Go traces of Boundary 0.5.1 in dev mode running on two different *BSD platforms that share the same PostgreSQL 13.x database on another OpenBSD-6.9 (stable) host:
a) FreeBSD-12.1 (actually OPNsense/HardenedBSD) for which I have cross-compiled Boundary dev with my small extension to write a trace file on OpenBSD-7.0 (snapshot) using Go 1.17 with XC_=OSARCH=freebsd/amd64 gmake dev (disabled ‘ui’ in Makefile).
Dev mode starts OK against the shared PostgreSQL 13 database on the remote OpenBSD host, and login from CLi is successful using Boundary binary from release site boundary authenticate password -auth-method-id=ampw_1234567890 -login-name=admin -password=password.
b) OpenBSD-7.0 (snapshot) on which have built Boundary dev with the same extension to write a trace file on OpenBSD-7.0 (snapshot) using Go 1.17.
Boundary dev started OK against the same shared remote PostgreSQL database, but appears to get stuck also on OpenBSD-7.0 with Go 1.17, like on OpenBSD-6.9 with Go 1.15.5 as described earlier.
Also, login from CLi fails with Error trying to perform authentication: error performing client request during Authenticate call: Post "http://127.0.0.1:9200/v1/auth-methods/ampw_1234567890:authenticate": EOF
After having isolated the shared PostgreSQL 13.x from the equation, I hope to narrow down why the same build of Boundary dev 0.5.1 on OpenBSD trace.out_OpenBSD.txt (1.4 MB)
gets “unresponsive” at least its API, but continues to output events occasionally, by comparing its traces against the known-good reference from FreeBSD trace.out_FreeBSD.txt (1.9 MB)
using go tool trace -http=127.0.0.1:6060 trace.out .
Things get very quiet on OpenBSD after the initial 20 secs
Using the same test setup on bare metal, I did a quick test with Boundary 0.4.0 and 0.3.0 in dev mode using binaries downloaded from the release site on OpenBSD-7.0 (snapshot) against Postgres 13.x on OpenBSD-6.9 (stable).
In both cases, Boundary dev mode appears to start OK, but login attempts from cli fail as well like with 0.5.1 on the same setup. Eventually after max_request_duration, they start to complain about
2021-09-05T10:31:04.842+0200 [INFO] worker: connected to controller: address=127.0.0.1:9201
2021-09-05T10:31:09.861+0200 [ERROR] worker: error making status request to controller: error=“rpc error: code = DeadlineExceeded desc = context deadline exceeded”
2021-09-05T10:31:16.691+0200 [ERROR] worker: error making status request to controller: error=“rpc error: code = DeadlineExceeded desc = context deadline exceeded”
…
Next, will revisit the limits of the login class for the Boundary user/daemon and crank them up, not only openfiles limit which had proven crucial earlier for Consul, Nomad and Vault.
