Trying mount CephFS volume into our Nomad cluster

Nomad
#1

I’m having a problem with the ceph-csi-plugin

labi@inspiron:~$ nomad status ceph-csi-plugin
ID            = ceph-csi-plugin
Name          = ceph-csi-plugin
Submit Date   = 2021-02-10T23:54:49+01:00
Type          = system
Priority      = 50
Datacenters   = dc1
Namespace     = default
Status        = running
Periodic      = false
Parameterized = false

Summary
Task Group  Queued  Starting  Running  Failed  Complete  Lost
nodes       0       0         4        4       16        0

Allocations
ID        Node ID   Task Group  Version  Desired  Status   Created    Modified
7f67a7d7  7cfca224  nodes       9        run      running  2h25m ago  11m2s ago
7fcf3e00  4aad1344  nodes       9        run      running  2h25m ago  11m2s ago
8e93d98a  5d1817df  nodes       9        run      running  2h25m ago  11m2s ago
a72ce57f  3508cce0  nodes       9        run      running  2h25m ago  11m2s ago

Then I run

nomad alloc logs -stderr 7f67a7d7 ceph-node

And this is the result

I0210 21:53:56.930562       1 cephcmds.go:53] ID: 461 Req-ID: c91a5d7e-6b1f-11eb-a64f-4201c0a8010b an error (exit status 1) and stdError (modprobe: FATAL: Module ceph not found in directory /lib/modules/3.10.0-1160.15.2.el7.x86_64
) occurred while running modprobe args: [ceph]
E0210 21:53:56.930590       1 nodeserver.go:175] ID: 461 Req-ID: c91a5d7e-6b1f-11eb-a64f-4201c0a8010b failed to mount volume c91a5d7e-6b1f-11eb-a64f-4201c0a8010b: an error (exit status 1) and stdError (modprobe: FATAL: Module ceph not found in directory /lib/modules/3.10.0-1160.15.2.el7.x86_64
#2

maybe a naive question, but are the CephFS packages installed cleanly (successfully) on the node?

1 Like
#3

Not sure what ‘cleanly’ means, but the Ceph gui tells me that the file system is healthy. What can I specifically check on my end to answer you more precisely?

#4 
[root@hashistack-client-0 ~]# yum list installed|grep ceph
ceph-common.x86_64                   2:15.2.8-0.el7                 @Ceph       
libcephfs2.x86_64                    2:15.2.8-0.el7                 @Ceph       
python3-ceph-argparse.x86_64         2:15.2.8-0.el7                 @Ceph       
python3-ceph-common.x86_64           2:15.2.8-0.el7                 @Ceph       
python3-cephfs.x86_64                2:15.2.8-0.el7                 @Ceph
#6

@shantanugadgil, This is what we have installed on the node. Is this what you were asking about?

#7

I was just trying to think of what could be possibly wrong, I am no Ceph expert! :innocent:

Any clue from lsmod | grep ceph ?

#8

From nomad client node:

[root@hashistack-client-0 wpfs]# lsmod |grep ceph
ceph                  363016  1 
libceph               306750  1 ceph
dns_resolver           13140  1 libceph
libcrc32c              12644  4 xfs,libceph,nf_nat,nf_conntrack

but from ceph-csi plugin container:

[root@a9f81dbe0549 /]# lsmod |grep ceph
ceph                  363016  1
libceph               306750  1 ceph
dns_resolver           13140  1 libceph
libcrc32c              12644  4 xfs,libceph,nf_nat,nf_conntrack
[root@a9f81dbe0549 /]# modprobe cephfs
modprobe: FATAL: Module cephfs not found in directory /lib/modules/3.10.0-1160.15.2.el7.x86_64
[root@a9f81dbe0549 /]# ls -al /lib/modules
total 0
drwxr-xr-x. 2 root root   6 May 11  2019 .
dr-xr-xr-x. 1 root root 188 Nov 19 13:06 ..
[root@a9f81dbe0549 /]#
#9

Does anyone have, at least, some good documentation for how to use the plugin?

#12

another thought: is there a way to mount the directory from the OS /lib/modules into the Docker container at the location /lib/modules.

Something like: /lib/modules:/lib/modules.

I haven’t played with CSI (with Nomad) so the above is also “just a thought” :slight_smile: :slight_smile: