Node with private LAN only - How to publish job with public IP of Server

Nomad
1

Motivation:
I want to host services which cannot be proxied by Caddy, which I intend to use as a reverse proxy.

For example, mumble should be reachable by a combo of publicip:port. Such services I intend to host on small-scale nodes connected to the server via a private network.

For now, I am also spinning up a client on the nomad server. Only the nomad server is connected to the Internet via a public IP.

Issue:
If I add a job, for example “Hello world” (I’m not much farther than this) to the client running on the server, it gets assigned the public IP and it works.

If I add the same job to the node, it just gets the private IP of the node.
I tried:

  • Adding the public IP as meta to the node and define in the job as address
  • Several attempts at different HTTP advertised addresses of the node: 10.0.0.1, public IP of server, 10.0.0.2
  • With bridge network mode and without

Every time, the allocation states: 10.0.0.3:port (for example, 80)

My Setup is the following:

Nomad server:

  • Public IP via eth0 78.54.xx.xx
  • Private IP via enp7s0 10.0.0.2
    Iptables rule to forward Internet traffic to connected clients:
    iptables -t nat -A POSTROUTING -s '10.0.0.0/16' -o eth0 -j MASQUERADE

Nomad client:

  • Private IP via enp7s0 10.0.0.3
  • Access to Internet via default route:
    ip route add default via 10.0.0.1

Internet “works”, meaning I can reach everything on the Internet

Overall:

  • I’m accessing the server via tailsacle and ssh to the node, since it’s not reachable via the internet

Server setup:

# Copyright (c) HashiCorp, Inc.
# SPDX-License-Identifier: MPL-2.0

# Full configuration options can be found at https://www.nomadproject.io/docs/configuration

data_dir  = "/home/infra/mnt/infra/services/nomad/data"
bind_addr = "0.0.0.0"
datacenter = "hydra"

addresses {
  http = "0.0.0.0"
  rpc  = "10.0.0.2"
  serf = "10.0.0.2"
}

advertise {
  # Defaults to the first private IP address.
  http = "0.0.0.0"
  rpc  = "10.0.0.2"
  serf = "10.0.0.2:5648" # non-default ports may be specified
}

limits {
  http_max_conns_per_client = 1000
}

server {
  # license_path is required for Nomad Enterprise as of Nomad v1.1.1+
  #license_path = "/etc/nomad.d/license.hclic"
  enabled          = true
  bootstrap_expect = 1
}

client {
  enabled = true
  cni_path = "/usr/libexec/cni"
  cni_config_dir = "/usr/libexec/cni"
}

Client setup:

# Copyright (c) HashiCorp, Inc.
# SPDX-License-Identifier: MPL-2.0

# Full configuration options can be found at https://www.nomadproject.io/docs/configuration

data_dir  = "/home/node/nomad"
datacenter = "hydra"

client {
  network_interface = "enp7s0"
  enabled = true
  cni_path = "/usr/libexec/cni"
  cni_config_dir = "/usr/libexec/cni"
  servers = ["10.0.0.2"]
}
2

I, kind of, managed to get where I wanted.

Since there’s no way to add an environmental variable from the host (such as export public_ip=xxxx), I went around that limitation:

  1. I created a dummy interface where set the public IP as its IPv4 address:
    PUBLIC_IP="$(curl icanhazip.com)"
    /sbin/ip link add eth0 type dummy
    /sbin/ip addr add $PUBLIC_IP dev eth0
    /sbin/ip link set eth0 up
    /sbin/ip route add default via 10.0.0.1

Then in the nomad.hcl I change the advertised http IP accordingly:

advertise {
  http = "{{ GetPublicIP }}"
  [...]
}

Every job thus is assigned to the public IP and I can reach it directly.
I don’t think this is a safe way to reach my initially stated motivation: To reach services which cannot be reverse proxies by caddy on nodes without direct Internet access, but it might do for now.

If there’s a better way to reach non http/https services with fixed ports, like mumble or game servers, running on nodes without direct Internet acces, please let me know.