CNI Network Not Found

Nomad
1

I’m tying to use the ipvlan cni plugin in a job but the job always reports that the network can’t be found.

client.driver_mgr.docker: failed to start container: driver=docker container_id=b7a91dd1c958606bb2f75cc3f803d1a0df9e415740a3bbf60a1e705716a13457 attempt=1 error="API error (404): network cni/pub not found"

Nomad detects the interface:

2024-02-25T15:53:11.359-0800 [DEBUG] client.fingerprint_mgr: detected CNI network: name=pub

But it always fails to find it.

Here’s my cni plugin config:

{
     "cniVersion": "0.4.0",
          "name": "pub",
          "disableCheck": true,
          "plugins": [
          {
               "type": "ipvlan",
               "master": "eth1",
               "ipam": {
                    "type": "static",
                    "addresses": 
                         [
                    {
                         "address": "10.10.30.150/24",
                         "gateway": "10.10.30.1"
                    }],

                    "routes" : [
                         { "dst": "0.0.0.0" }
                    ],
                    "dns": {
                         "nameservers" : ["10.10.30.121"],
                         "domain": "example.com",
                         "search": [ "example.com" ]
                    }
               }
          },
        {
          "type": "portmap",
          "capabilities": { "portMappings": true },
          "snat": true
        }

     ]
}

I’ve tried specifying the cni network in my job in multiple ways:

group "test" {
    count = 1

    network {
      mode = "cni/pub"

And like this:

    task "test-ipvlan" {
      driver = "docker"
      config {
        network_mode = "cni/pub"

It does work if I create the pub network manually using ‘docker network create’.

This happens with any cni network or network type that I define (ipvlan, bridge, dhcp are the ones that I’ve tried).

I’m pretty sure that I’m missing something pretty simple here.

2

Ensure your CNI config file is in the correct directory (/opt/cni/config), CNI plugins are installed in /opt/cni/bin, and your Nomad client is configured to use the right network interface. In your job spec, use mode = "cni/pub" under the network stanza at the group level, not under config for the Docker driver. If the issue persists, check Nomad logs for more insights and verify CNI plugin compatibility with your Nomad version.

3

My config is in /opt/cni/config and the plugins are in /opt/cni/bin. I might have been unclear in what I tried. I did try configuring it in the network stanza like this:

group "test" {
    count = 1

    network {
      mode = "cni/pub"

And get this error:

failed to setup alloc: pre-run hook "network" failed: failed to create network for alloc

It’s clear that nomad IS seeing the cni networks so I don’t think it’s an issue with the cni paths. From nomad debug logs:

2024-02-25T15:53:11.359-0800 [DEBUG] client.fingerprint_mgr: detected CNI network: name=pub

I’m not sure where to go from here.

4

Solved. Apparently this was actually an issue grabbing the pause container. I repointed to my internal registry by setting infra_image in my nomad config.

Thanks for the assistance!