How to change unique.network.ip-address for a node?

Hey all! I’m really stumped atm with jobs being bound to the wrong address.

Really fast, my network is setup like this: each of my servers has 2 IP addresses, the local one assigned to them by whatever network they are on and then their IP addresses on my VPN.

In the past, all jobs were correctly scheduled onto the VPN addresses and everything worked fine. The other day though, I added a new client node onto the cluster and nomad has decided to use the local address for new nodes, and this seems to be because it has set the local address as the unique.network.ip-address on this node. However, I’m not sure what I should do to fix this!

Originally, the bind address in my nomad.hcl was 0.0.0.0, but the issue persists with the bind address changed to the VPN address. Changing the bind_addr in the nomad.hcl does change the advertise address, but not the unique ip-address field that seems to be corelated with container bind addresses.

What do I need to do to change what address jobs are bound to? Do you want me to send any more information? Thank you!!

Problematic client nomad.hcl:

datacenter = "home"

data_dir = "/opt/nomad/data"
bind_addr = "#####"  # <-- the VPN address of this node

telemetry {
  collection_interval = "1s"
  use_node_name = true
  prometheus_metrics = true
  publish_allocation_metrics = true
  publish_node_metrics = true
}

client {
  enabled = true
  servers = ["#####:4647"]  # <-- VPN address of the nomad server

  host_volume "traefik" {
    path = "/opt/nomad/traefik"
    read_only = false
  }
  host_volume "torrent-data" {
    path      = "/data-external/Torrents"
    read_only = false
  }

  host_volume "torrent-config" {
    path      = "/opt/nomad/torrent-config"
    read_only = false
  }
}

consul {
    address = "127.0.0.1:8500"
    client_service_name = "#####"
    auto_advertise      = true
    client_auto_join    = true
}

plugin "docker" { }

Ok I got it, there’s a client option called network_interface. No idea how I missed that!!!

2 Likes

network_interface doesn’t seem to work for me, unique.network.ip-address still evaluates to the local address. In templates I can use GetInterfaceIp "<iface>", but that only works in certain contexts. Tried using nomad.advertise.address which gets the ip from bind_addr, but that variable includes the nomad port as well for some reason, and there is no way to strip that with variable interpolation… So I’m a bit stuck, any pointers?

Hi @exFalso. Thanks for using Nomad!

For your use case, does using either ${attr.unique.network.ip-address} or ${NOMAD_IP_<label>} give you the expected value? Note, you’d have to replace <label> with the port label name.

${attr.unique.network.ip-address} resolves to the local non-VPN IP, even though network_interface is set to the VPN interface.

NOMAD_IP_.. is an env variable only available inside the task… but I checked it and that is set to the right IP, so that could be used inside a task.

Unfortunately we need that IP in other places as well, e.g. when setting up DNS for a container in docker (dns_servers), or sometimes in extra_hosts. I found workarounds for each of these specific situations for now, but still haven’t found an answer to OP’s question.