I’ve spun up a fresh K8s cluster on DigitalOcean, and I’ve gone through the setup steps for HCP Waypoint, including installing the runner in the cluster.
All of the resources in the cluster are healthy. However, the Waypoint runner isn’t receiving any tasks. All jobs are stuck in the QUEUED state and all tasks are stuck in the PENDING state. Reading the logs from the runner’s pod, I get…
» Runner configuration:
Server address: api.hashicorp.cloud:443
Type: remote
» Runner logs:
2023-05-04T19:27:25.351Z [INFO] waypoint.runner.agent: initializing the runner
2023-05-04T19:27:25.351Z [INFO] waypoint.runner.agent.runner: state directory does not exist, creating it: state_dir=/data/runner
2023-05-04T19:27:25.354Z [INFO] waypoint.runner.agent: starting runner: id=01GZM3Q9H5RYRGH2GVTTG22BW3
2023-05-04T19:27:26.926Z [INFO] waypoint.runner.agent.runner.config_recv: new configuration received
2023-05-04T19:27:27.427Z [INFO] waypoint.runner.agent.runner: runner registered with server and ready
2023-05-04T19:27:27.427Z [INFO] waypoint.runner.agent.runner: accepting jobs concurrently: count=3
2023-05-04T19:27:27.428Z [INFO] waypoint.runner.agent.runner: waiting for job assignment
2023-05-04T19:27:27.428Z [INFO] waypoint.runner.agent.runner: waiting for job assignment
2023-05-04T19:27:27.428Z [INFO] waypoint.runner.agent.runner: waiting for job assignment
2023-05-04T19:27:27.878Z [INFO] waypoint.runner.agent.runner.config_recv: new configuration received
2023-05-04T19:29:34.859Z [INFO] waypoint.runner.agent.runner.config_recv: new configuration received
2023-05-04T19:29:44.879Z [INFO] waypoint.runner.agent.runner.config_recv: new configuration received
2023-05-04T19:29:51.884Z [INFO] waypoint.runner.agent.runner.config_recv: new configuration received
2023-05-04T19:33:01.894Z [INFO] waypoint.runner.agent.runner.config_recv: new configuration received
2023-05-04T19:33:36.918Z [ERROR] waypoint.runner.agent.runner: error running job: error="rpc error: code = Internal desc = server closed the stream without sending trailers"
2023-05-04T19:33:36.919Z [INFO] waypoint.runner.agent.runner: waiting for job assignment
2023-05-04T19:33:36.919Z [ERROR] waypoint.runner.agent.runner.config_recv: error receiving configuration, exiting: err="rpc error: code = Internal desc = server closed the stream without sending trailers"
2023-05-04T19:33:37.378Z [ERROR] waypoint.runner.agent.runner: error running job: error="rpc error: code = Internal desc = server closed the stream without sending trailers"
2023-05-04T19:33:37.378Z [INFO] waypoint.runner.agent.runner: waiting for job assignment
2023-05-04T19:33:37.378Z [ERROR] waypoint.runner.agent.runner: error running job: error="rpc error: code = Internal desc = server closed the stream without sending trailers"
2023-05-04T19:33:37.378Z [INFO] waypoint.runner.agent.runner: waiting for job assignment
This implies to me that either the HCP Waypoint server is sending requests incorrectly or not at all. I think “not at all” is more likely since the log file hasn’t grown within the last hour.
It’s not clear to me how to debug this further, since everything is healthy and this is a fresh cluster and a fresh HCP Waypoint project; nothing is obviously out of the ordinary. I’m only trying to run any kind of “hello world” project.