Firewall rules for waypoint runner

I am using ufw on Linux to lock down the host and run Nomad on different IP than the default (so it can run on my Tailscale network). Waypoint Server works fine but the Waypoint Runner fails to start. If I disable the firewall, the runner starts properly.

What are the required firewall rules for Waypoint Runner installed into Nomad?

Or alternatively, is this a bug where Waypoint Runner can’t exist with these restrictive firewall rules but Waypoint Server can?

ufw

Default rule is deny.

$ sudo ufw status
Status: active

To                         Action      From
--                         ------      ----
Anywhere on tailscale0     ALLOW       Anywhere
41641/udp                  ALLOW       Anywhere
Anywhere (v6) on tailscale0 ALLOW       Anywhere (v6)
41641/udp (v6)             ALLOW       Anywhere (v6)

Allocation logs

|Jul 20, '21 07:24:32 +0100|Alloc Unhealthy|Unhealthy because of failed task|
|---|---|---|
|Jul 20, '21 07:24:32 +0100|Not Restarting|Exceeded allowed attempts 2 in interval 30m0s and mode is fail|
|Jul 20, '21 07:24:32 +0100|Terminated|Exit Code: 1, Exit Message: Docker container exited with non-zero exit code: 1|
|Jul 20, '21 07:24:26 +0100|Started|Task started by client|
|Jul 19, '21 23:10:31 +0100|Driver|Downloading image|
|Jul 19, '21 23:10:15 +0100|Restarting|Task restarting in 15.474501718s|
|Jul 19, '21 23:10:15 +0100|Terminated|Exit Code: 1, Exit Message: Docker container exited with non-zero exit code: 1|
|Jul 19, '21 23:10:09 +0100|Started|Task started by client|
|Jul 19, '21 23:10:06 +0100|Driver|Downloading image|
|Jul 19, '21 23:09:50 +0100|Restarting|Task restarting in 16.629469115s|

stderr

2021-07-19T22:09:44.451Z [INFO]  waypoint: waypoint version: full_string="v0.4.1 (4d5c838e+CHANGES)" version=v0.4.1 prerelease= metadata= revision=4d5c838e+CHANGES
2021-07-19T22:09:44.451Z [TRACE] waypoint: starting interrupt listener for context cancellation
2021-07-19T22:09:44.452Z [TRACE] waypoint: interrupt listener goroutine started
2021-07-19T22:09:44.456Z [DEBUG] waypoint: home configuration directory: path=/home/waypoint/.config/waypoint
2021-07-19T22:09:44.456Z [TRACE] waypoint: no API client provided, initializing connection if possible
2021-07-19T22:09:44.457Z [INFO]  waypoint.server: attempting to source credentials and connect
2021-07-19T22:09:49.457Z [ERROR] waypoint: failed to create client: error="context deadline exceeded"
2021-07-19T22:09:49.457Z [TRACE] waypoint: stopping signal listeners and cancelling the context
2021-07-19T22:10:09.789Z [INFO]  waypoint: waypoint version: full_string="v0.4.1 (4d5c838e+CHANGES)" version=v0.4.1 prerelease= metadata= revision=4d5c838e+CHANGES
2021-07-19T22:10:09.790Z [TRACE] waypoint: starting interrupt listener for context cancellation
2021-07-19T22:10:09.791Z [TRACE] waypoint: interrupt listener goroutine started
2021-07-19T22:10:09.795Z [DEBUG] waypoint: home configuration directory: path=/home/waypoint/.config/waypoint
2021-07-19T22:10:09.795Z [TRACE] waypoint: no API client provided, initializing connection if possible
2021-07-19T22:10:09.795Z [INFO]  waypoint.server: attempting to source credentials and connect
2021-07-19T22:10:14.798Z [ERROR] waypoint: failed to create client: error="context deadline exceeded"
2021-07-19T22:10:14.798Z [TRACE] waypoint: stopping signal listeners and cancelling the context
2021-07-20T06:24:26.974Z [INFO]  waypoint: waypoint version: full_string="v0.4.1 (4d5c838e+CHANGES)" version=v0.4.1 prerelease= metadata= revision=4d5c838e+CHANGES
2021-07-20T06:24:26.975Z [TRACE] waypoint: starting interrupt listener for context cancellation
2021-07-20T06:24:26.976Z [TRACE] waypoint: interrupt listener goroutine started
2021-07-20T06:24:26.978Z [DEBUG] waypoint: home configuration directory: path=/home/waypoint/.config/waypoint
2021-07-20T06:24:26.979Z [TRACE] waypoint: no API client provided, initializing connection if possible
2021-07-20T06:24:26.980Z [INFO]  waypoint.server: attempting to source credentials and connect
2021-07-20T06:24:31.981Z [ERROR] waypoint: failed to create client: error="context deadline exceeded"
2021-07-20T06:24:31.982Z [TRACE] waypoint: stopping signal listeners and cancelling the context

With the firewall disabled, the runner starts fine:

» Runner configuration:
  Server address: 100.103.178.62:28239


» Runner logs:

Hi davidmytton,

The runner needs to be able to egress the waypoint server via GRPC. From your log, it looks like your runner is attempting to reach your server at 100.103.178.62:28239, and is failing because of your restrictive firewall. You will likely want to enable ingress to the waypoint server in any case, so opening this ip/port in UFW should solve the problem.

If your runner and server are on the same machine on the host network, and your server is also bound to the loopback interface, you can configure the runner to connect to the server at 127.0.0.1:28239 instead of <public-ip>:28239, bypassing ufw.

You may also not need the runner - it is currently only used for builds and deployments triggered by git changes, so if you don’t intend to enable the git integration you can safely live without it for the time being.

Let me know if that works for you!

Izaak

Opening the port on ufw worked, although the port changed from when I first posted this thread. With waypoint v0.5.2 (20d177b6+CHANGES) it is now 29360. Is it expected that the port will change?

For a nomad server installation, waypoint currently uses a dynamic port and the dynamic allocation IP, so each will change with every server install, and beware: if the nomad allocation moves (i.e. due to an client restart), the address and port will change again. You can track this issue here: `waypoint install` on nomad service discovery problems · Issue #2314 · hashicorp/waypoint · GitHub

In the meantime, if you need a stable server address, you can create your own nomad job using the waypoint docker container (hashicorp/waypoint), the waypoint server run command, which will allow you to configure a stable consul service.