CSI: pre-run hook "csi_hook" failed

Nomad
1

Hello:

I did have a working environment with rocketDuck’s NFS CSI plugin, but I wanted to try out the latest “official” K8s plugin. So far, the controller and nodes are up. The volume was registered successfully, but when attempting to run a job against the CSI plugin, I am getting the following:
Error:
Setup Failure: failed to setup alloc: pre-run hook “csi_hook” failed: mounting volumes: rpc error: code = InvalidArgument desc = server is a required parameter

I am using the following:
Nomad: 1.8.1
csi-driver-nfs: 4.7.0
Podman: 5.1.1

Is anyone using the latest NFS plugin without issues? The error is vague to me; the only thing I can think of is the server value in the volume declaration (which is correct).

Any help would be grateful.

Thanks

Controller

job "csi-nfs-plugin-controller" {
  datacenters = [ "lab" ]

  group "controller" {
    task "plugin" {
      driver = "podman"

      config {
        image = "registry.k8s.io/sig-storage/nfsplugin:v4.7.0"
        
        args = [
          "--endpoint=unix://csi/csi.sock",
          "--nodeid=${attr.unique.hostname}",
          "--drivername=nfs.csi.k8s.io",
          "--logtostderr",
          "--v=5",
        ]
      }

      csi_plugin {
        id             = "nomad-csi-nfs"
        type           = "controller"
        mount_dir      = "/csi"
        health_timeout = "150s"
      }

      resources {
        cpu    = 64
        memory = 128
      }
    }
  }
}

Storage nodes:

job "csi-nfs-plugin-nodes" {
  datacenters = [ "lab" ]
  type        = "system"

  group "nodes" {
    task "plugin" {
      driver = "podman"

      config {
        image = "registry.k8s.io/sig-storage/nfsplugin:v4.7.0"

        args = [
          "--endpoint=unix://csi/csi.sock",
          "--nodeid=${attr.unique.hostname}",
          "--drivername=nfs.csi.k8s.io",
          "--logtostderr",
          "--v=5",
        ]

        network_mode = "host"
        privileged   = true
      }

      csi_plugin {
        id        = "nomad-csi-nfs"
        type      = "node"
        mount_dir = "/csi"
      }

      resources {
        memory = 128
      }
    }
  }
}

Volume declaration:

type      = "csi"
id        = "netbootxyz"
name      = "netbootxyz"
plugin_id = "nomad-csi-nfs"
capability {
  access_mode     = "multi-node-multi-writer"
  attachment_mode = "file-system"
}

// Parameters: https://github.com/kubernetes-csi/csi-driver-nfs/blob/master/docs/driver-parameters.md
parameters {
  server           = "192.168.108.12"
  share            = "/mnt/pool02/csi/"
  mountPermissions = "0"
}

mount_options {
  fs_type     = "nfs"
  mount_flags = ["hard", "timeo=30", "intr", "vers=4", "_netdev", "nolock"]
}

Application:

job "netbootxyz" {
  datacenters = ["lab"]
  type        = "service"

  group "netbootxyz" {
    network {
      mode = "cni/netbootxyz"
      port "ui" {
        static = 3000
      }
      port "tftp" {
        static = 69
      }
    }

    reschedule {
      attempts       = 3
      delay          = "20s"
      delay_function = "exponential"
      interval       = "3m"
      unlimited      = false
    }

    volume "netbootxyz" {
      type            = "csi"
      source          = "netbootxyz"
      read_only       = false
      attachment_mode = "file-system"
      access_mode     = "multi-node-multi-writer"
    }

    task "netbootxyz" {
      driver = "podman"

      config {
        image      = "netbootxyz/netbootxyz:0.7.1-nbxyz3"
        ports      = ["ui", "tftp"]
        privileged = true
      }

      volume_mount {
        volume      = "netbootxyz"
        destination = "/assets"
        read_only   = false
      }

      volume_mount {
        volume      = "netbootxyz"
        destination = "/config"
        read_only   = false
      }

      resources {
        cpu        = 1000
        memory     = 1024
      }
    }
  }
}
2

I found my problem over on Github.

Changing from “parameters” to “context” no longer prevents the allocation from spinning up.

While my original problem was fixed, the allocation has no persistent storage (i.e. no folder on the NFS server). Let me dig into this, but most likely, will open an issue with @thatsk if I get stuck.

Thanks for looking…