How to configure CNI Macvlan in Nomad

Nomad
1

Dear board members,

I’m trying to configure macvlan using CNI in Nomad.

Currently I have got it working via Docker networking (created a macvlan network on each nomad client):

docker network create -d macvlan
–subnet=192.168.137.0/24
–gateway=192.168.137.1
-o parent=eth0 ingress

Then using this job config will launch a docker container on a nomad client which is reachable from the outside with the specified IP address, moving the container around will also move the IP to the new host which is what I need:

job “docs” {
datacenters = [“dc1”]

group “example” {
count = 1

network {
port “http” {
to = “8181”
}
}

task “server” {
driver = “docker”

 service {
   name = "http-echo"
   port = "http"
   address_mode = "driver"
   check {
     name     = "http-echo-check"
     type     = "tcp"
     interval = "10s"
     timeout  = "2s"
   }
 }

 resources {
   cpu    = 600
   memory = 128
 }

 config {
   image = "hashicorp/http-echo"
   network_mode = "ingress"
   ipv4_address = "192.168.137.232"
   ports = ["http"]
   args = [
     "-listen",
     ":8181",
     "-text",
     "hello world",
   ]
 }

}
}
}

However, this is the docker way, not the Nomad CNI way… I did configure a macvlan interface via CNI (/opt/cni/config/ingress.conflist):

{
“cniVersion”: “0.4.0”,
“name”: “ingress”,
“plugins”: [
{
“type”: “macvlan”,
“master”: “eth0”,
“ipam”: {
“type”: “host-local”,
“subnet”: “192.168.137.224/27”,
“gateway”: “192.168.137.1”
}
},
{
“type”: “portmap”,
“capabilities”: { “portMappings”: true },
“snat”: true
}
]
}

However when I use this by adding this into the network stanza of the job config above Nomad won’t assign the IP to the docker container…

mode = “cni/ingress”

I’ve been looking everywhere online but could not find a good help resource. I hope that someone here is able to help out :slight_smile:

1 Like
2

This is a real issue. I have been struggling for days to setup a working network with cni with no luck.
There are no resources as to exactly how to make it work. For example with the same setup as above,

  • instead of defining the service stanza in the task with driver,
  • define it in the group with alloc

it actually advertizes to consule an IP that has been seamingly been allocated from the subnet,

However, that is not case. The advertizment does not match the actual IP given to the container.

3

The following CNI config and job stanza will configure the docker container to use an IP within the CNI network, in this case an ipvlan network but it should also work with other CNI plugins.

Sadly, it does not adhere to the IP address included within the ipv4_address specification. It will just pick an random available IP. Within the range.

However, consul does have the correct IP on which the container is reachable. Please mind that for the service discovery to work the service checks must pass. As the local consul client will perform the checks those will fail because ipvlan and macvlan are designed that way (host cannot reach the macvlan interfaces and vise versa). If you want to make that work you need to add another ipvlan/macvlan on your host network with an IP within the range.

Now, I’m wondering. If the mac address feature does work, we might be able to change the CNI ipam config to use a DHCP server (hosted on one of the nomad masters for example), and create a static lease there using the specified mac address.

Note that the DHCP hack won’t work on any hosted service: Azure, AWS, Hetzner, etc. MacVlan won’t work there as well, that’s why I switched to IPVlan…

CNI config:

{
  "cniVersion": "0.4.0",
  "name": "ingress",
  "plugins": [
	{
	  "type": "ipvlan",
	  "master": "ens32",
	  "ipam": {
		"type": "host-local",
		"ranges": [
			[
				{
					"subnet": "192.168.1.0/24",
					"rangeStart": "192.168.1.224",
					"rangeEnd": "192.168.1.254",
					"gateway": "192.168.1.1"
				}
			]
		]
	  }
	},
	{
	  "type": "portmap",
	  "capabilities": { "portMappings": true },
	  "snat": true
	}
  ]
}

Nomad Job stanza:

job "docs" {
	datacenters = ["dc1"]

	group "example" {
		count = 1

		network {
			mode = "cni/ingress"
		}

		service {
			address_mode = "alloc"
			port = "8181"
			task = "http-echo"
			name = "echo-example"
			check {
				name     = "http-echo-check"
				type     = "http"
				protocol = "http"
				path     = "/"
				interval = "3s"
				timeout  = "1s"
			}
		}

		task "http-echo" {
			driver = "docker"
				
			config {
				image = "hashicorp/http-echo"
				ipv4_address = "192.168.1.232"
				args = [
					"-listen",
					":8181",
					"-text",
					"hello world",
				]
			}
		}
	}
}
4

After a few hours I’ve gave up again on CNI as I can’t get it to work with a static IP.

However, this can work when using the docker network stack as stated above.

For others who want to have a static IP on a docker container within Nomad, which is very handy for ingress containers:

Create the Docker IPVlan network:

docker network create -d ipvlan  \
	--subnet=192.168.1.0/24  \
	--ip-range=192.168.1.224/27 \
	--gateway=192.168.1.1 \
	-o parent=ens32 ingress

Create the interface to enable the host to communicate with the local docker containers:

ip link add ingress-host link ens32 type ipvlan mode l3
ip addr add 192.168.1.221/32 dev ingress-host
ip link set ingress-host up
ip route add 192.168.1.224/27 dev ingress-host

Nomad example job:

job "docs" {
	datacenters = ["lab"]

	group "example" {
		count = 1

		task "http-echo" {
			driver = "docker"

			config {
				image = "hashicorp/http-echo"
				network_mode = "ingress"
				ipv4_address = "192.168.1.232"
				args = [
					"-listen",
					":8181",
					"-text",
					"hello world",
				]
			}

			service {
				name = "echo-example"
				address_mode = "driver"
				port = "8181"

				check {
					name		 = "http-echo-check"
					type		 = "http"
					protocol	 = "http"
					path		 = "/"
					interval	 = "3s"
					timeout		 = "1s"
					address_mode = "driver"
				}
			}
		}
	}
}

Please mind that the network stanza within the group part isn’t required. But if you do add it, Nomad will assign a random dynamic port and within Nomad it will look like if the task is running on the host IP which it isn’t.

You can ping the IP assigned to the container from the host:

# ping 192.168.1.232
PING 192.168.1.232 (192.168.1.232) 56(84) bytes of data.
64 bytes from 192.168.1.232: icmp_seq=1 ttl=64 time=0.039 ms

In Consul you will see the correct IP:

# dig @192.168.1.201 -p 8600 echo-example.service.lab.consul +short
192.168.1.232
# dig SRV @192.168.1.201 -p 8600 echo-example.service.lab.consul +short
1 1 8181 c0a801e8.addr.lab.consul.
# dig @192.168.1.201 -p 8600 c0a801e8.addr.lab.consul +short
192.168.1.232

If someone can get the static IP working with CNI IPVlan/MacVlan then I would appreciate it if you can share the configuration with me

3 Likes