Help with csi driver, ceph and configmaps


we are running a ceph cluster and we are evaluating moving from kubernetes to nomad.

I’ve setup a nomad cluster, and I’m trying to use the ceph-csi driver.

I tried to follow the documentation

I’ve created this job:

job "ceph-csi-nodes" {
  datacenters = ["dc1"]

  # you can run node plugins as service jobs as well, but this ensures
  # that all nodes in the DC have a copy.
  type = "system"

  group "nodes" {
    task "plugin" {
      driver = "docker"

      config {
        image = ""

        args = [

        privileged = true

      env {
        "ClusterID" = "<my_cluster_id>"
        "pool" = "SSDPool"

      csi_plugin {
        id        = "ceph-rdb"
        type      = "node"
        mount_dir = "/csi"

      resources {
        cpu    = 500
        memory = 256

Output from nomad job status:

ID              Type     Priority  Status   Submit Date
ceph-csi-nodes  system   50        running  2020-04-22T09:13:47+02:100:

Output from nomad plugin status ceph-rdb:

ID                   = ceph-rdb
Provider             =
Version              = v2.1.0
Controllers Healthy  = 0
Controllers Expected = 0
Nodes Healthy        = 1
Nodes Expected       = 1

ID        Node ID   Task Group  Version  Desired  Status   Created     Modified
480b6015  61b2567b  nodes       3        run      running  56m36s ago  22s ago

I’ve created a volume configuration like this one:

id = "ssd-volume"
name = "ssd volume"
type = "csi"
external_id = "<ceph_cluster_id>"
plugin_id = "ceph-rdb"
access_mode = "single-node-writer"
attachment_mode = "file-system"
mount_options {
   fs_type = "ext4"
   mount_flags = ["rw"]

Note: <ceph_cluster_id> is the value got from ceph fsid

Output from nomad volume status:

Container Storage Interface
ID        Name        Plugin ID  Schedulable  Access Mode
ssd-volu  ssd volume  ceph-rdb   true         single-node-writer

If I try to create a job running that volume I get this error:

failed to setup alloc: pre-run hook "csi_hook" failed: rpc error: code = InvalidArgument desc = stage secrets cannot be nil or empty

Because I guess I need to point to the Ceph monitors and configure the cluster to let the plugin know how to interact with ceph.

In kubernetes this is done with a ConfigMap. Check this example.

My question is how can I provide that kind of configuration to the plugin. There is no such concept of ConfigMaps in Nomad afaik.

I know this is a new feature and that the ceph-csi plugin only supports kubernetes officially, but I understand it should work with nomad too.

If I can figure out how to make it work I’d be happy to write a tutorial or docs about it.

Any help is very appreciated!

1 Like

Hi @AdrianRibao! We need to implement the secrets field in the CSI spec to make Ceph work. That didn’t ship in the beta for 0.11.0, but it’s looking to be one of the first post-beta features we’re working on for CSI. Take a look at and please feel free to drop a report in there (it helps me convince the folks internally that this is important to get done!)

Thanks @tgross! Do you mean to paste what I wrote here in the ticket?

Sure, that’d help! Thanks!