Guide to SMB-CSI Driver

Nomad
1

As per the title - are there any guides or samples on how to use the SMB-CSI driver?

kubernetes-csi/csi-driver-smb: This driver allows Kubernetes to access SMB Server on both Linux and Windows nodes. (github.com)

Thanks.

2

Hey :wave:

If you simply want to consume existing SMB/CIFS shares it’s actually rather simple.

You need to run a controller job, a node plugin job (on every node where SMB volumes should be accessible to jobs) and then create the relevant volume config files and register then in your cluster.

The node plugin must be run as a privileged container - if you are on Nomad Enterprise you can use sentinel to enforce that only certain jobs are allowed to have this option set :

A policy such as this could be used as inspiration :

# in allowed namespaces.
main = rule { deny_priviledged_containers() }

deny_priviledged_containers = func() {
    # list namespaces that priviledged containers are allowed to run in
    allowed_namespaces = "platform"

    # loop over task groups and run check
    for job.task_groups as tg {
        for tg.tasks as task {
         priviledged_requested = task.config.privileged else "false"
          if priviledged_requested == "true" and job.namespace not in allowed_namespaces {
            return false
          } else {
            return true
          }
        }
    }
}

I have attached the 3 job definitions as inspiration :slight_smile:

HCL file for running the controller :

job "csi-smb-controller" {
  datacenters = ["DC-A", "DC-B"]
  namespace = "platform"

  constraint {
    attribute = "${node.class}"
    value     = "compute"
  }

  spread {
    attribute = "${node.datacenter}"
    weight    = 100
  }
  
  group "controller" {
    count = 2
    
    task "plugin" {
      driver = "docker"

      config {
        image = "mcr.microsoft.com/k8s/csi/smb-csi:v1.7.0"
        args = [
          "--v=5",
          "--nodeid=${attr.unique.hostname}",
          "--endpoint=unix:///csi/csi.sock",
          "--drivername=smb.csi.k8s.io"
        ]
      }

      csi_plugin {
        id        = "smb"
        type      = "controller"
        mount_dir = "/csi"
      }

      resources {
        memory = 512
        cpu    = 512
      }
    } // task
  } // group
} // job

The node plugin which has to run on any node where you want CSI-SMB volumes to be accessible :

job "plugin-smb-nodes" {
  datacenters = ["DC-A", "DC-B"]
  type        = "system"
  namespace   = "platform"

  constraint {
    attribute = "${node.class}"
    value     = "compute"
  }
 
  group "nodes" {
    
    task "plugin" {
      driver = "docker"

      config {
        image = "mcr.microsoft.com/k8s/csi/smb-csi:v1.7.0"
        args = [
          "--v=5",
          "--nodeid=${attr.unique.hostname}",
          "--endpoint=unix:///csi/csi.sock",
          "--drivername=smb.csi.k8s.io"
        ]
        # node plugins must run as privileged jobs because they
        # mount disks to the host
        privileged = true
      }

      csi_plugin {
        id        = "smb"
        type      = "node"
        mount_dir = "/csi"
      }

      resources {
        memory = 512
        cpu    = 512
      }
    }
  }
}

Example volume config :

plugin_id   = "smb"
type        = "csi"
id          = "EXAMPLE_SHARE_NAME"
name        = "EXAMPLE_SHARE_NAME"
external_id = "EXAMPLE_SHARE_NAME"

capability {
  access_mode = "multi-node-multi-writer"
  attachment_mode = "file-system"
}

secrets {
  username = "user@domain"
  password = "whatever"
}

context {
  source = "//server/share$"
  subDir = "somefolder/someother_folder"
}

mount_options {
  mount_flags = [ "dir_mode=0777", "file_mode=0777", "vers=3.0", "nounix", "noserverino" ]
}
3 Likes
3

Thank you so much, that was exactly what I was looking for - still struggling with mapping k8s syntax to Nomad.

4

… once the controller and node plugins are running and you have created a volume a job definition for consuming said volume could look something like this :

job "share-test" {
  datacenters = ["DC-A", "DC-B"]
  type        = "service"

  group "share-test" {

    volume "EXAMPLE_SHARE_NAME" {
      type            = "csi"
      source          = "EXAMPLE_SHARE_NAME"
      access_mode     = "multi-node-multi-writer"
      attachment_mode = "file-system"
    }

    task "share-test" {
      driver = "docker"

      # add volume to container
      volume_mount {
        volume      = "EXAMPLE_SHARE_NAME"
	    destination = "/share"
      }

      config {
       image = "ubuntu:22.10"
        args  = ["sleep", "infinity"]
      }

      resources {
        cpu    = 500
        memory = 256
      }

    } // task ends here
  }   // group ends here
}     // job ends here
1 Like
5

Thank you!

Currently watching my cluster rebuild :slight_smile: but will try this out, much better than manually creating cifs mounts on all nodes.

6

Hi! I applied this code on Windows nodes, but I have an error. Do you know how can I fix it?
Driver Failure failed to create container: API error (400): invalid mount config for type "bind": invalid mount path: '/csi'

7

If you’re having problems with adding a volume that uses this SMB solution you might want to change the context into parameters:

plugin_id   = "smb"
type        = "csi"
id          = "EXAMPLE_SHARE_NAME"
name        = "EXAMPLE_SHARE_NAME"
external_id = "EXAMPLE_SHARE_NAME"

capability {
  access_mode = "multi-node-multi-writer"
  attachment_mode = "file-system"
}

secrets {
  username = "user@domain"
  password = "whatever"
}

parameters {
  source = "//server/share$"
  subDir = "somefolder/someother_folder"
}

mount_options {
  mount_flags = [ "dir_mode=0777", "file_mode=0777", "vers=3.0", "nounix", "noserverino" ]
}
8

Where can you find the reference to the image? I’m looking into updating the plugin, but wasn’t able to find a working reference.