Custom CNI network

Nomad
1

I’m experimenting with a single-node instance of Nomad (v1.2.0). This client is using the podman driver. It already had a bridge (br0) before I started using Nomad, and I’m using the CNI option for networking. The conflist looks like this:

{
  "cniVersion": "0.4.0",
  "name": "testing",
  "plugins": [
    {
      "type": "loopback"
    },
    {
      "type": "bridge",
      "bridge": "br0",
      "isGateway": false,
      "ipMasq": false,
      "hairpinMode": false,
      "ipam": { 
        "type": "host-local",
        "routes": [{ "dst": "0.0.0.0/0" }],
        "ranges": [
          [
            {
              "subnet": "192.168.1.0/24",
              "rangeStart": "192.168.1.160",
              "rangeEnd": "192.168.1.191",
              "gateway": "192.168.1.1"
            }
          ]
        ]
      }
    }
  ]
}

The intent is to allocate IP addresses directly on my network to each job’s group and not do port mapping.

In an example job’s configuration, my group section includes

network {
  mode = "cni/testing"
  port "http" {
    static = "80"
  }
}

and

service {
  tags = ["http"]
  name = "${JOB}"
  port = "http"
  address_mode = "alloc"
}

I am able to create nomad jobs and access the created containers through the allocated IP addresses. The IP addresses are also being correctly published in Consul. In general it seems to be working.

However, the web ui for Nomad shows allocations as having a host address of e.g. 192.168.1.10:80 with a mapped port of 0. This is showing my client’s address. I would expect to see the IP that is being published in Consul. Also (perhaps due to the same root issue), if I plan another job with a static port of 80, I get the error:

“network: reserved port collision http=80” exhausted on 1 nodes

This is unexpected because the port would be allocated on a new IP address. There shouldn’t be any conflict with the host’s network.

Any clues as to what I might be doing wrong?

Thanks!

2

Hi @nil and thanks for this detailed question.

CNI is only responsible for configuring the network namespace in which the container runs and doesn’t assign and configure IP addresses and interfaces on the host. In this specific conflist, you are using bridge mode which means CNI will ensure a veth is connected between the host bridge interface and the container network namespace interface.

In order to correctly route from the host network into the container network namespace, IP tables will be used generally with dynamic port assignments. The use of static ports will result in the port conflict error if more than one allocation is triggered on a single host.

To see how CNI configures the network namespace, you could use the nomad alloc exec command to exec into a container and iterate the network.

I hope this helps; please feel free to ask any follow-up questions you might have.

Thanks,
jrasell and the Nomad team.

3

Thanks for the reply @jrasell.

In this use case where I’m using an existing bridge on the host (which is already connected to the LAN), iptables is not required for network connectivity from the LAN to the container. The container is bridged directly to the LAN. The jobs I’ve scheduled are accessible to other hosts on the LAN, so it’s definitely working.

From what I could see with CNI, it returns (among other information) the address assigned from its host-local ipam. I was hoping Nomad might have a way to only use that returned address and not do anything with NAT through iptables.

It does appear that Nomad is using that address when it publishes to Consul, but not within Nomad itself. I haven’t peeked at the Nomad source to see how this is handled.

I’m guessing this is just not an expected usage case for Nomad. But maybe it could be considered?