Help with csi driver, ceph and configmaps

Nomad
1

Hello.

we are running a ceph cluster and we are evaluating moving from kubernetes to nomad.

I’ve setup a nomad cluster, and I’m trying to use the ceph-csi driver.

I tried to follow the documentation https://learn.hashicorp.com/nomad/stateful-workloads/csi-volumes

I’ve created this job:

job "ceph-csi-nodes" {
  datacenters = ["dc1"]

  # you can run node plugins as service jobs as well, but this ensures
  # that all nodes in the DC have a copy.
  type = "system"

  group "nodes" {
    task "plugin" {
      driver = "docker"

      config {
        image = "quay.io/cephcsi/cephcsi:v2.1.0"

        args = [
            "--nodeid=${node.unique.name}",
            "--type=rbd",
            "--nodeserver=true",
            "--endpoint=unix://csi/csi.sock",
            "--v=5",
            "--drivername=rbd.csi.ceph.com",
        ]

        privileged = true
      }

      env {
        "ClusterID" = "<my_cluster_id>"
        "pool" = "SSDPool"
      }

      csi_plugin {
        id        = "ceph-rdb"
        type      = "node"
        mount_dir = "/csi"
      }

      resources {
        cpu    = 500
        memory = 256
      }
    }
  }
}

Output from nomad job status:

ID              Type     Priority  Status   Submit Date
ceph-csi-nodes  system   50        running  2020-04-22T09:13:47+02:100:

Output from nomad plugin status ceph-rdb:

ID                   = ceph-rdb
Provider             = rbd.csi.ceph.com
Version              = v2.1.0
Controllers Healthy  = 0
Controllers Expected = 0
Nodes Healthy        = 1
Nodes Expected       = 1

Allocations
ID        Node ID   Task Group  Version  Desired  Status   Created     Modified
480b6015  61b2567b  nodes       3        run      running  56m36s ago  22s ago

I’ve created a volume configuration like this one:

id = "ssd-volume"
name = "ssd volume"
type = "csi"
external_id = "<ceph_cluster_id>"
plugin_id = "ceph-rdb"
access_mode = "single-node-writer"
attachment_mode = "file-system"
mount_options {
   fs_type = "ext4"
   mount_flags = ["rw"]
}

Note: <ceph_cluster_id> is the value got from ceph fsid

Output from nomad volume status:

Container Storage Interface
ID        Name        Plugin ID  Schedulable  Access Mode
ssd-volu  ssd volume  ceph-rdb   true         single-node-writer

If I try to create a job running that volume I get this error:

failed to setup alloc: pre-run hook "csi_hook" failed: rpc error: code = InvalidArgument desc = stage secrets cannot be nil or empty

Because I guess I need to point to the Ceph monitors and configure the cluster to let the plugin know how to interact with ceph.

In kubernetes this is done with a ConfigMap. Check this example.

My question is how can I provide that kind of configuration to the plugin. There is no such concept of ConfigMaps in Nomad afaik.

I know this is a new feature and that the ceph-csi plugin only supports kubernetes officially, but I understand it should work with nomad too.

If I can figure out how to make it work I’d be happy to write a tutorial or docs about it.

Any help is very appreciated!

1 Like
2

Hi @AdrianRibao! We need to implement the secrets field in the CSI spec to make Ceph work. That didn’t ship in the beta for 0.11.0, but it’s looking to be one of the first post-beta features we’re working on for CSI. Take a look at https://github.com/hashicorp/nomad/issues/7668 and please feel free to drop a report in there (it helps me convince the folks internally that this is important to get done!)

3

Thanks @tgross! Do you mean to paste what I wrote here in the ticket?

4

Sure, that’d help! Thanks!