is is possible to run a ceph cluster inside of nomad the same way as in a Kubernetes cluster and if so any documentation to do this? Im 30% done moving from a k3s cluster but volume and backups have kept me here for some time till I can wrap my head around it.
Thanks
Not really my ideal solution but I used cephadm to deploy the cluster outside of nomad , that works just wish I could figure out to have nomad control it as well
So I think I should be able to create new volume from nomad using the nomad volume create, but following Commands: volume create | Nomad by HashiCorp does not see to work.
First of I had to go into the ceph controller and mkdir -p /tmp/csi/keys/ as it could not create a tempt keyfile and now its failing to see monitors?
$ nomad volume create mysql-volume.hcl
Error creating volume: Unexpected response code: 500 (rpc error: 1 error occurred:
* controller create volume: CSI.ControllerCreateVolume: volume "mysql_data" snapshot source &{"" ""} is not compatible with these parameters: rpc error: code = InvalidArgument desc = failed to fetch monitor list using clusterID (efddd514-caf9-11eb-a54a-8be5f459e004): missing configuration for cluster ID "efddd514-caf9-11eb-a54a-8be5f459e004"
)
I0613 06:33:09.293467 1 utils.go:166] ID: 11573 Req-ID: mysql_data GRPC request: {"accessibility_requirements":{},"capacity_range":{"limit_bytes":50000000000,"required_bytes":50000000000},"name":"mysql_data","parameters":{"clusterID":"efddd514-caf9-11eb-a54a-8be5f459e004","imageFeatures":"layering","pool":"nomadv"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":1}}]}
E0613 06:33:09.294052 1 rbd_util.go:959] ID: 11573 Req-ID: mysql_data failed getting mons (failed to fetch monitor list using clusterID (efddd514-caf9-11eb-a54a-8be5f459e004): missing configuration for cluster ID "efddd514-caf9-11eb-a54a-8be5f459e004")
E0613 06:33:09.294234 1 utils.go:171] ID: 11573 Req-ID: mysql_data GRPC error: rpc error: code = InvalidArgument desc = failed to fetch monitor list using clusterID (efddd514-caf9-11eb-a54a-8be5f459e004): missing configuration for cluster ID "efddd514-caf9-11eb-a54a-8be5f459e004"
id = "ceph_mysql_data"
name = "mysql_data"
type = "csi"
plugin_id = "ceph-csi"
capacity_max = "50G"
capacity_min = "50G"
capability {
access_mode = "single-node-writer"
attachment_mode = "file-system"
}
secrets {
userID = "myPoolAdmin"
userKey = "SNIPPED"
}
parameters {
clusterID = "efddd514-caf9-11eb-a54a-8be5f459e004"
pool = "nomadv"
imageFeatures = "layering"
}
mount_options {
fs_type = "ext4"
}
also as a sidenote I get Error decoding the volume definition: At 23:17: unknown slice type: *ast.LiteralType when the mount options have anything set
I have not made any progress on this still but very much interested in completing the hashistack life. I dont think there is a better supported CSI implementation but open to suggestion to use other than portworx or ceph if that is what is needed.
Thanks!
So doing without dynamic volumes, is there a working example of ceph volume hcl registration with out using the create command?
I don’t know much about Ceph, but have you seen the demo here? Maybe there’s something in there that can help you.
Hey, thanks for pointing me in that direction! I have seen that and have given that a shot in the past but as I have been learning and adapting my setup to work may have had other variables going, So reset and started from the beginning with that readme without other references and still running into issues.
$ nomad volume create ./patroni-volume.hcl
Error creating volume: Unexpected response code: 500 (rpc error: 1 error occurred:
* controller create volume: rpc error: controller create volume: CSI.ControllerCreateVolume: volume "patroni-volume" snapshot source &{"" ""} is not compatible with these parameters: rpc error: code = InvalidArgument desc = failed to fetch monitor list using clusterID (efddd514-caf9-11eb-a54a-8be5f459e004): missing configuration for cluster ID "efddd514-caf9-11eb-a54a-8be5f459e004"