Multi-Interface Networking in Nomad 0.12 not working

Nomad
1

Hello,

I am experimenting with the new Multi-Interface Networking in Nomad 0.12 but I can’t get it to work. I am sure I am doing something wrong, but I have no idea what it is.
I have a Linux-host with to network-interfaces (eth0 and eth1). The internal network is connected at eth0 and the public network (internet) is connected at eth1.
IP-addresses at eth1 are assigned through keepalived.

I have added the following snippet to the client.hcl for Nomad:

client {
    ...
    host_network "internet_test" {
        cidr = "12.13.14.15/32"  #I replaced the real public IP for a fake one in this snippet for obvious reasons
        interface = "eth1"
    }
}

I have restarted the Nomad-service to apply these settings.

My job is defined as follows:

job "nginx-test" {
  datacenters = ["mydc"]
  type = "system"

  update {
    max_parallel = 1
    stagger = "10s"
  }

  group "containers" {
    restart {
      interval = "10m"
      attempts = 3
      delay    = "15s"
      mode     = "delay"
    }

    volume "html" {
      type = "host"
      read_only = true
      source = "html-files"
    }

    task "nginx-test" {
      service {
        name = "nginx-test"
        port = "http"
        check {
          type     = "http"
          protocol = "http"
          method   = "GET"
          header {
            Host = [ "test.myhost.nl" ]
          }
          port     = "http"
          path     = "/health"
          tls_skip_verify = true
          interval = "1s"
          timeout  = "1s"
        }
      }

      template {
        data = <<EOF
server {
    listen 80;
    server_name test.myhost.nl;

    location / {
      root html;
    }

    location /health {
      return 200 'OK';
    }

    location /test {
      return 200 'OK';
    }
}
EOF
        destination = "local/conf.d/default.conf"
        change_mode = "signal"
        change_signal = "SIGHUP"
      }

      driver = "docker"

      config {
        image = "nginx:1.19.0-alpine-perl"

        volumes = [
          "local/conf.d/:/etc/nginx/conf.d/"
        ]

        port_map {
          http = 80
        }
      }

      env {
        TZ = "Europe/Amsterdam"
      }

      volume_mount {
        volume = "html"
        destination = "/etc/nginx/html/"
      }

      resources {
        memory = 300
        network {
          mode = "bridge"

          port "http" {
            static = "80"
            host_network = "internet_test"
          }
        }
      }
    }
  }
}

After running this job, the Docker container spins up. However, it still binds to my internal network instead of my external IP-adres.
Output of docker ps:

CONTAINER ID    IMAGE                                   PORTS
1de7e975f72a     nginx:1.19.0-alpine-perl    10.10.80.211:80->80/tcp, 10.10.80.211:80->80/udp

I was expecting an output like:

CONTAINER ID    IMAGE                                   PORTS
1de7e975f72a     nginx:1.19.0-alpine-perl    12.13.14.15:80->80/tcp, 12.13.14.15:80->80/udp

Am I missing something or am I misunderstanding the new multi-interface networking feature of Nomad 0.12?

Any help is appreciated

2

I opened an issue for this here: https://github.com/hashicorp/nomad/issues/8432

3

Hey @HumanPrinter

Sorry for the delayed response here. I’m working with a few folks with a similar issue in the link @Legogris posted above.

In short there’s some validation missing here to catch that you’re trying to use host networks with task based networks. Host networks are only supported with group based networks in bridge networking mode. See the documentation here: https://www.nomadproject.io/docs/job-specification/network#host-networks

If you move your network stanza to the group level and let nomad handle port mapping with the bridge networking mode you should see the traffic forwarding correctly.

I’m working on flushing out some additional documentation and docker specific examples that should be available soon.

2 Likes