# Ceph CSI.ControllerCreateVolume connecting failed: rados: ret=-13, Permission denied

**URL:** <https://discuss.hashicorp.com/t/ceph-csi-controllercreatevolume-connecting-failed-rados-ret-13-permission-denied/43437>\
**Category:** Nomad\
**Tags:** csi\
**Created:** [August 22, 2022, 8:34pm UTC](https://discuss.hashicorp.com/t/ceph-csi-controllercreatevolume-connecting-failed-rados-ret-13-permission-denied/43437 "2022-08-22T20:34:52Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![nomar.mora](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/nomar.mora/32/27268_2.png) [@nomar.mora](https://discuss.hashicorp.com/u/nomar.mora)\
**Post date:** [August 22, 2022, 8:34pm UTC](https://discuss.hashicorp.com/t/ceph-csi-controllercreatevolume-connecting-failed-rados-ret-13-permission-denied/43437/1 "2022-08-22T20:34:52Z")

</div>

Hi!  
Installed nomad and then installed ceph following this guides:

[https://docs.ceph.com/en/latest/cephadm/](https://docs.ceph.com/en/latest/cephadm/)  
[https://docs.ceph.com/en/latest/rbd/rbd-nomad/](https://docs.ceph.com/en/latest/rbd/rbd-nomad/)

```auto
root@server01:/# ceph --version
ceph version 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) quincy (stable)

root@server01:/home/opr# nomad --version
Nomad v1.3.3 (428b2cd8014c48ee9eae23f02712b7219da16d30)

```

Created a pool named nomad and run the rbd command, as the guide said.

I runned this command inside `cephadm shell` command:

```auto
ceph auth get-or-create client.nomad mon 'profile rbd' osd 'profile rbd pool=nomad' mgr 'profile rbd pool=nomad'

```

Run this command on nomad clients:

```auto
modprobe rbd

```

And restarted nomad

Then I run this jobs files based on guide files:

```auto
job "ceph-csi-plugin-controller" {
  datacenters = ["dc1"]
  group "controller" {
    network {
      port "metrics" {}
    }
    task "ceph-controller" {
      template {
        data = <<EOF
[{
    "clusterID": "ddeb11aa-223c-11ed-8020-4fb209447fe5",
    "monitors": [
        "10.0.0.10",
        "10.0.0.11",
        "10.0.0.12",
        "10.0.0.20",
        "10.0.0.21"
    ]
}]
EOF
        destination = "local/config.json"
        change_mode = "restart"
      }
      driver = "docker"
      config {
        image = "quay.io/cephcsi/cephcsi:v3.3.1"
        volumes = [
          "./local/config.json:/etc/ceph-csi-config/config.json"
        ]
        mounts = [
          {
            type = "tmpfs"
            target = "/tmp/csi/keys"
            readonly = false
            tmpfs_options = {
              size = 1000000 # size in bytes
            }
          }
        ]
        args = [
          "--type=rbd",
          "--controllerserver=true",
          "--drivername=rbd.csi.ceph.com",
          "--endpoint=unix://csi/csi.sock",
          "--nodeid=${node.unique.name}",
          "--instanceid=${node.unique.name}-controller",
          "--pidlimit=-1",
          "--logtostderr=true",
          "--v=5",
          "--metricsport=$${NOMAD_PORT_metrics}"
        ]
      }
      resources {
        cpu = 500
        memory = 256
      }
      service {
        name = "ceph-csi-controller"
        port = "metrics"
        tags = ["prometheus"]
      }
      csi_plugin {
        id = "ceph-csi"
        type = "controller"
        mount_dir = "/csi"
      }
    }
  }
}

```

And

```auto
job "ceph-csi-plugin-nodes" {
  datacenters = ["dc1"]
  type = "system"
  group "nodes" {
    network {
      port "metrics" {}
    }
    task "ceph-node" {
      driver = "docker"
      template {
        data = <<EOF
[{
    "clusterID": "ddeb11aa-223c-11ed-8020-4fb209447fe5",
    "monitors": [
        "10.0.0.10",
        "10.0.0.11",
        "10.0.0.12",
        "10.0.0.20",
        "10.0.0.21"
    ]
}]
EOF
        destination = "local/config.json"
        change_mode = "restart"
      }
      config {
        image = "quay.io/cephcsi/cephcsi:v3.3.1"
        volumes = [
          "./local/config.json:/etc/ceph-csi-config/config.json"
        ]
        mounts = [
          {
            type = "tmpfs"
            target = "/tmp/csi/keys"
            readonly = false
            tmpfs_options = {
              size = 1000000 # size in bytes
            }
          }
        ]
        args = [
          "--type=rbd",
          "--drivername=rbd.csi.ceph.com",
          "--nodeserver=true",
          "--endpoint=unix://csi/csi.sock",
          "--nodeid=${node.unique.name}",
          "--instanceid=${node.unique.name}-nodes",
          "--pidlimit=-1",
          "--logtostderr=true",
          "--v=5",
          "--metricsport=$${NOMAD_PORT_metrics}"
        ]
        privileged = true
      }
      resources {
        cpu = 500
        memory = 256
      }
      service {
        name = "ceph-csi-nodes"
        port = "metrics"
        tags = ["prometheus"]
      }
      csi_plugin {
        id = "ceph-csi"
        type = "node"
        mount_dir = "/csi"
      }
    }
  }
}

```

At last when I try to run this:

```auto
id = "ceph-mysql"
name = "ceph-mysql"
type = "csi"
plugin_id = "ceph-csi"
capacity_max = "8G"
capacity_min = "5G"

capability {
  access_mode = "single-node-writer"
  attachment_mode = "file-system"
}

secrets {
  userID = "admin"
  userKey = "AQDPzQNjqhZSARAAIDku3bLDEleVNvpScQMHPw=="
}

parameters {
  clusterID = "ddeb11aa-223c-11ed-8020-4fb209447fe5"
  pool = "nomad"
  imageFeatures = "layering"
}

```

I got this message:

```auto
Error creating volume: Unexpected response code: 500 (rpc error: rpc error: 1 error occurred:
	* controller create volume: CSI.ControllerCreateVolume: controller plugin returned an internal error, check the plugin allocation logs for more information: rpc error: code = Internal desc = failed to get connection: connecting failed: rados: ret=-13, Permission denied

)

```

I tried changing “admin” with “client.nomad” at clusterID with the same result.

Please any ideas?

Thanks in advance  
Nomar

---

<div class="post-metadata">

**Author:** ![nomar.mora](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/nomar.mora/32/27268_2.png) [@nomar.mora](https://discuss.hashicorp.com/u/nomar.mora)\
**Post date:** [August 22, 2022, 9:34pm UTC](https://discuss.hashicorp.com/t/ceph-csi-controllercreatevolume-connecting-failed-rados-ret-13-permission-denied/43437/2 "2022-08-22T21:34:16Z")

</div>

🤦🏼‍♂️

The guide was wrong! The correct volume creation was:

```auto
id = "ceph-mysql"
name = "ceph-mysql"
type = "csi"
plugin_id = "ceph-csi"
capacity_max = "8G"
capacity_min = "5G"

capability {
  access_mode = "single-node-writer"
  attachment_mode = "file-system"
}

secrets {
  userID = "nomad"
  userKey = "AQDPzQNjqhZSARAAIDku3bLDEleVNvpScQMHPw=="
}

parameters {
  clusterID = "ddeb11aa-223c-11ed-8020-4fb209447fe5"
  pool = "nomad"
  imageFeatures = "layering"
}

```

On userID the correct value is nomad, not admin as the guide says not client.nomad as I was using, just nomad.

I realizes the truth reading this article:

> **[Provision Volumes on Kubernetes and Nomad using Ceph CSI](https://itnext.io/provision-volumes-from-external-ceph-storage-on-kubernetes-and-nomad-using-ceph-csi-7ad9b15e9809)**
>
> If you want to run stateful applications on Kubernetes, CSI(Container Storage Interface) has an important role in provisioning volumes…