# Help with csi driver, ceph and configmaps

**URL:** <https://discuss.hashicorp.com/t/help-with-csi-driver-ceph-and-configmaps/7933>\
**Category:** Nomad\
**Created:** [April 22, 2020, 8:21am UTC](https://discuss.hashicorp.com/t/help-with-csi-driver-ceph-and-configmaps/7933 "2020-04-22T08:21:09Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![AdrianRibao](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/adrianribao/32/6200_2.png) [@AdrianRibao](https://discuss.hashicorp.com/u/AdrianRibao)\
**Post date:** [April 22, 2020, 8:21am UTC](https://discuss.hashicorp.com/t/help-with-csi-driver-ceph-and-configmaps/7933/1 "2020-04-22T08:21:09Z")

</div>

Hello.

we are running a ceph cluster and we are evaluating moving from kubernetes to nomad.

I’ve setup a nomad cluster, and I’m trying to use the [ceph-csi](https://github.com/ceph/ceph-csi) driver.

I tried to follow the documentation [https://learn.hashicorp.com/nomad/stateful-workloads/csi-volumes](https://learn.hashicorp.com/nomad/stateful-workloads/csi-volumes)

I’ve created this job:

```auto
job "ceph-csi-nodes" {
  datacenters = ["dc1"]

  # you can run node plugins as service jobs as well, but this ensures
  # that all nodes in the DC have a copy.
  type = "system"

  group "nodes" {
    task "plugin" {
      driver = "docker"

      config {
        image = "quay.io/cephcsi/cephcsi:v2.1.0"

        args = [
            "--nodeid=${node.unique.name}",
            "--type=rbd",
            "--nodeserver=true",
            "--endpoint=unix://csi/csi.sock",
            "--v=5",
            "--drivername=rbd.csi.ceph.com",
        ]

        privileged = true
      }

      env {
        "ClusterID" = "<my_cluster_id>"
        "pool" = "SSDPool"
      }

      csi_plugin {
        id = "ceph-rdb"
        type = "node"
        mount_dir = "/csi"
      }

      resources {
        cpu = 500
        memory = 256
      }
    }
  }
}

```

Output from `nomad job status`:

```auto
ID Type Priority Status Submit Date
ceph-csi-nodes system 50 running 2020-04-22T09:13:47+02:100:

```

Output from `nomad plugin status ceph-rdb`:

```auto
ID = ceph-rdb
Provider = rbd.csi.ceph.com
Version = v2.1.0
Controllers Healthy = 0
Controllers Expected = 0
Nodes Healthy = 1
Nodes Expected = 1

Allocations
ID Node ID Task Group Version Desired Status Created Modified
480b6015 61b2567b nodes 3 run running 56m36s ago 22s ago

```

I’ve created a volume configuration like this one:

```auto
id = "ssd-volume"
name = "ssd volume"
type = "csi"
external_id = "<ceph_cluster_id>"
plugin_id = "ceph-rdb"
access_mode = "single-node-writer"
attachment_mode = "file-system"
mount_options {
   fs_type = "ext4"
   mount_flags = ["rw"]
}

```

_Note: `<ceph_cluster_id>` is the value got from `ceph fsid`_

Output from `nomad volume status`:

```auto
Container Storage Interface
ID Name Plugin ID Schedulable Access Mode
ssd-volu ssd volume ceph-rdb true single-node-writer

```

If I try to create a job running that volume I get this error:

```auto
failed to setup alloc: pre-run hook "csi_hook" failed: rpc error: code = InvalidArgument desc = stage secrets cannot be nil or empty

```

Because I guess I need to point to the Ceph monitors and configure the cluster to let the plugin know how to interact with ceph.

In kubernetes this is done with a ConfigMap. [Check this example](https://github.com/ceph/ceph-csi/blob/master/examples/csi-config-map-sample.yaml).

My question is how can I provide that kind of configuration to the plugin. There is no such concept of ConfigMaps in Nomad afaik.

I know this is a new feature and that the ceph-csi plugin only supports kubernetes officially, but I understand it should work with nomad too.

If I can figure out how to make it work I’d be happy to write a tutorial or docs about it.

Any help is very appreciated!

---

<div class="post-metadata">

**Author:** ![tgross](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/tgross/32/3015_2.png) [@tgross](https://discuss.hashicorp.com/u/tgross)\
**Post date:** [April 22, 2020, 9:04pm UTC](https://discuss.hashicorp.com/t/help-with-csi-driver-ceph-and-configmaps/7933/2 "2020-04-22T21:04:31Z")

</div>

Hi @AdrianRibao! We need to implement the secrets field in the CSI spec to make Ceph work. That didn’t ship in the beta for 0.11.0, but it’s looking to be one of the first post-beta features we’re working on for CSI. Take a look at [https://github.com/hashicorp/nomad/issues/7668](https://github.com/hashicorp/nomad/issues/7668) and please feel free to drop a report in there (it helps me convince the folks internally that this is important to get done!)

---

<div class="post-metadata">

**Author:** ![AdrianRibao](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/adrianribao/32/6200_2.png) [@AdrianRibao](https://discuss.hashicorp.com/u/AdrianRibao)\
**Post date:** [April 23, 2020, 9:49am UTC](https://discuss.hashicorp.com/t/help-with-csi-driver-ceph-and-configmaps/7933/3 "2020-04-23T09:49:14Z")

</div>

Thanks @tgross! Do you mean to paste what I wrote here in the ticket?

---

<div class="post-metadata">

**Author:** ![tgross](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/tgross/32/3015_2.png) [@tgross](https://discuss.hashicorp.com/u/tgross)\
**Post date:** [April 23, 2020, 3:30pm UTC](https://discuss.hashicorp.com/t/help-with-csi-driver-ceph-and-configmaps/7933/4 "2020-04-23T15:30:50Z")

</div>

Sure, that’d help! Thanks!