Ceph CSI.ControllerCreateVolume connecting failed: rados: ret=-13, Permission denied

Nomad
1

Hi!
Installed nomad and then installed ceph following this guides:

https://docs.ceph.com/en/latest/cephadm/
https://docs.ceph.com/en/latest/rbd/rbd-nomad/

root@server01:/# ceph --version
ceph version 17.2.3 (dff484dfc9e19a9819f375586300b3b79d80034d) quincy (stable)

root@server01:/home/opr# nomad --version
Nomad v1.3.3 (428b2cd8014c48ee9eae23f02712b7219da16d30)

Created a pool named nomad and run the rbd command, as the guide said.

I runned this command inside cephadm shell command:

ceph auth get-or-create client.nomad mon 'profile rbd' osd 'profile rbd pool=nomad' mgr 'profile rbd pool=nomad'

Run this command on nomad clients:

modprobe rbd

And restarted nomad

Then I run this jobs files based on guide files:

job "ceph-csi-plugin-controller" {
  datacenters = ["dc1"]
  group "controller" {
    network {
      port "metrics" {}
    }
    task "ceph-controller" {
      template {
        data        = <<EOF
[{
    "clusterID": "ddeb11aa-223c-11ed-8020-4fb209447fe5",
    "monitors": [
        "10.0.0.10",
        "10.0.0.11",
        "10.0.0.12",
        "10.0.0.20",
        "10.0.0.21"
    ]
}]
EOF
        destination = "local/config.json"
        change_mode = "restart"
      }
      driver = "docker"
      config {
        image = "quay.io/cephcsi/cephcsi:v3.3.1"
        volumes = [
          "./local/config.json:/etc/ceph-csi-config/config.json"
        ]
        mounts = [
          {
            type     = "tmpfs"
            target   = "/tmp/csi/keys"
            readonly = false
            tmpfs_options = {
              size = 1000000 # size in bytes
            }
          }
        ]
        args = [
          "--type=rbd",
          "--controllerserver=true",
          "--drivername=rbd.csi.ceph.com",
          "--endpoint=unix://csi/csi.sock",
          "--nodeid=${node.unique.name}",
          "--instanceid=${node.unique.name}-controller",
          "--pidlimit=-1",
          "--logtostderr=true",
          "--v=5",
          "--metricsport=$${NOMAD_PORT_metrics}"
        ]
      }
      resources {
        cpu    = 500
        memory = 256
      }
      service {
        name = "ceph-csi-controller"
        port = "metrics"
        tags = [ "prometheus" ]
      }
      csi_plugin {
        id        = "ceph-csi"
        type      = "controller"
        mount_dir = "/csi"
      }
    }
  }
}

And

job "ceph-csi-plugin-nodes" {
  datacenters = ["dc1"]
  type        = "system"
  group "nodes" {
    network {
      port "metrics" {}
    }
    task "ceph-node" {
      driver = "docker"
      template {
        data        = <<EOF
[{
    "clusterID": "ddeb11aa-223c-11ed-8020-4fb209447fe5",
    "monitors": [
        "10.0.0.10",
        "10.0.0.11",
        "10.0.0.12",
        "10.0.0.20",
        "10.0.0.21"
    ]
}]
EOF
        destination = "local/config.json"
        change_mode = "restart"
      }
      config {
        image = "quay.io/cephcsi/cephcsi:v3.3.1"
        volumes = [
          "./local/config.json:/etc/ceph-csi-config/config.json"
        ]
        mounts = [
          {
            type     = "tmpfs"
            target   = "/tmp/csi/keys"
            readonly = false
            tmpfs_options = {
              size = 1000000 # size in bytes
            }
          }
        ]
        args = [
          "--type=rbd",
          "--drivername=rbd.csi.ceph.com",
          "--nodeserver=true",
          "--endpoint=unix://csi/csi.sock",
          "--nodeid=${node.unique.name}",
          "--instanceid=${node.unique.name}-nodes",
          "--pidlimit=-1",
          "--logtostderr=true",
          "--v=5",
          "--metricsport=$${NOMAD_PORT_metrics}"
        ]
        privileged = true
      }
      resources {
        cpu    = 500
        memory = 256
      }
      service {
        name = "ceph-csi-nodes"
        port = "metrics"
        tags = [ "prometheus" ]
      }
      csi_plugin {
        id        = "ceph-csi"
        type      = "node"
        mount_dir = "/csi"
      }
    }
  }
}

At last when I try to run this:

id           = "ceph-mysql"
name         = "ceph-mysql"
type         = "csi"
plugin_id    = "ceph-csi"
capacity_max = "8G"
capacity_min = "5G"

capability {
  access_mode     = "single-node-writer"
  attachment_mode = "file-system"
}

secrets {
  userID  = "admin"
  userKey = "AQDPzQNjqhZSARAAIDku3bLDEleVNvpScQMHPw=="
}

parameters {
  clusterID     = "ddeb11aa-223c-11ed-8020-4fb209447fe5"
  pool          = "nomad"
  imageFeatures = "layering"
}

I got this message:

Error creating volume: Unexpected response code: 500 (rpc error: rpc error: 1 error occurred:
	* controller create volume: CSI.ControllerCreateVolume: controller plugin returned an internal error, check the plugin allocation logs for more information: rpc error: code = Internal desc = failed to get connection: connecting failed: rados: ret=-13, Permission denied

)

I tried changing “admin” with “client.nomad” at clusterID with the same result.

Please any ideas?

Thanks in advance
Nomar

2

:man_facepalming:t3:

The guide was wrong! The correct volume creation was:

id           = "ceph-mysql"
name         = "ceph-mysql"
type         = "csi"
plugin_id    = "ceph-csi"
capacity_max = "8G"
capacity_min = "5G"

capability {
  access_mode     = "single-node-writer"
  attachment_mode = "file-system"
}

secrets {
  userID  = "nomad"
  userKey = "AQDPzQNjqhZSARAAIDku3bLDEleVNvpScQMHPw=="
}

parameters {
  clusterID     = "ddeb11aa-223c-11ed-8020-4fb209447fe5"
  pool          = "nomad"
  imageFeatures = "layering"
}

On userID the correct value is nomad, not admin as the guide says not client.nomad as I was using, just nomad.

I realizes the truth reading this article:

1 Like