Question: how to run task in multi-interface configuration with access to docker network?

Nomad
1

Hi,

I would like to run container on nomad cluster, that will have access to multiple network interfaces and custom docker network.

I have the test job with config:

job "test" {
  datacenters = ["dc1"]
  type        = "service"

  group "test" {
    count = 1

    network {
      mode = "bridge"

      port "http" {
        host_network = "public"
        static = 8001
        to = 8000
      }
    }

    service {
      address_mode = "host"
      name         = "test"
      port         = "http"
    }

    task "test" {
      driver = "docker"

      config {
        image = "python:3.8-alpine"
        # network_mode = "test"
        args  = ["python", "-m", "http.server"]
      }
    }
  }
}

client config:

data_dir = "/var/lib/nomad"

log_level = "DEBUG"

client {
  enabled = 1
  servers = ["127.0.0.1"]

  network_interface = "eth0"

  host_network "public" {
    cidr = "192.168.88.0/24"
    interface = "eth1"
  }
}


plugin "docker" {
  config {
    volumes {
      enabled = true
    }
  }
}

server {
  bootstrap_expect = 1
  enabled = 1
  server_join {
    retry_join = ["127.0.0.1"]
  }
}

(also installed CNI plugins in /opt/cni/bin path)

As you see, default network interface is eth0 and I’d like to have my task test exposed only on eth1.

# ip -br a
lo               UNKNOWN        127.0.0.1/8 ::1/128
eth0             UP             192.168.88.250/24 fe80::215:5dff:fe0b:ac0f/64
eth1             UP             192.168.88.248/24 fe80::cf0d:e1a4:47ca:2036/64
docker0          DOWN           172.17.0.1/16 fe80::42:93ff:fea0:c016/64
nomad            UP             172.26.64.1/20 fe80::5042:c1ff:fe01:8283/64
br-517e1d2d528e  UP             172.19.0.1/16 fe80::42:e8ff:fe51:3ae4/64

So far it works:

# curl -I 192.168.88.248:8001
HTTP/1.0 200 OK
Server: SimpleHTTP/0.6 Python/3.8.7
Date: Wed, 10 Feb 2021 09:56:37 GMT
Content-type: text/html; charset=utf-8
Content-Length: 1030

Now I want to access another container via docker network from task “test”, so I uncomment line network_mode = "test" in job definition.

Let’s run new container:

docker run --name test2 -d --network test python:3.8-alpine python3 -m http.server

After deploying new container I have access to custom container called test2:

# docker exec -it test-1ab49ded-e7c2-109a-ddbd-4fa8506fb499 curl test2:8000 -I
HTTP/1.0 200 OK
Server: SimpleHTTP/0.6 Python/3.8.7
Date: Wed, 10 Feb 2021 10:00:59 GMT
Content-type: text/html; charset=utf-8
Content-Length: 915=

But it’s not possible to get access to this node from public network

# curl 192.168.88.248:8001
curl: (7) Failed to connect to 192.168.88.248 port 8001: Connection refused

In alloc status I see 192.168.88.248:8001 -> 8000

nomad alloc status 1ab49ded
ID                  = 1ab49ded-e7c2-109a-ddbd-4fa8506fb499
Eval ID             = 3cda7537
Name                = test.test[0]
Node ID             = 351e050e
Node Name           = centos8.localdomain
Job ID              = test
Job Version         = 1
Client Status       = running
Client Description  = Tasks are running
Desired Status      = run
Desired Description = <none>
Created             = 13m3s ago
Modified            = 12m46s ago
Deployment ID       = 3294659a
Deployment Health   = healthy

Allocation Addresses (mode = "bridge")
Label  Dynamic  Address
*http  yes      192.168.88.248:8001 -> 8000

Task "test" is "running"
Task Resources
CPU        Memory           Disk     Addresses
0/100 MHz  9.6 MiB/300 MiB  300 MiB

Task Events:
Started At     = 2021-02-10T09:58:55Z
Finished At    = N/A
Total Restarts = 0
Last Restart   = N/A

Recent Events:
Time                       Type        Description
2021-02-10T10:58:55+01:00  Started     Task started by client
2021-02-10T10:58:54+01:00  Task Setup  Building Task Directory
2021-02-10T10:58:48+01:00  Received    Task received by client

versions:

# nomad -v
Nomad v1.0.3 (08741d9f2003ec26e44c72a2c0e27cdf0eadb6ee)

# uname -a
Linux centos8.localdomain 4.18.0-193.19.1.el8_2.x86_64 #1 SMP Mon Sep 14 14:37:00 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

# docker -v
Docker version 19.03.13, build 4484c46d9d

# cat /etc/centos-release
CentOS Linux release 8.2.2004 (Core)

And the question is: what am I doing wrong?

2

I have asked the same question in github, it was closed today: Multi-Interface doesn't work if using network_mode stanza in docker driven task · Issue #10010 · hashicorp/nomad · GitHub

1 Like