# Internal Ceph Cluster

**URL:** <https://discuss.hashicorp.com/t/internal-ceph-cluster/25275>\
**Category:** Nomad\
**Created:** [June 10, 2021, 3:27am UTC](https://discuss.hashicorp.com/t/internal-ceph-cluster/25275 "2021-06-10T03:27:04Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![NegativeFeedback](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/negativefeedback/32/18743_2.png) [@NegativeFeedback](https://discuss.hashicorp.com/u/NegativeFeedback)\
**Post date:** [June 10, 2021, 3:27am UTC](https://discuss.hashicorp.com/t/internal-ceph-cluster/25275/1 "2021-06-10T03:27:05Z")

</div>

is is possible to run a ceph cluster inside of nomad the same way as in a Kubernetes cluster and if so any documentation to do this? Im 30% done moving from a k3s cluster but volume and backups have kept me here for some time till I can wrap my head around it.

Thanks

---

<div class="post-metadata">

**Author:** ![NegativeFeedback](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/negativefeedback/32/18743_2.png) [@NegativeFeedback](https://discuss.hashicorp.com/u/NegativeFeedback)\
**Post date:** [June 12, 2021, 12:36am UTC](https://discuss.hashicorp.com/t/internal-ceph-cluster/25275/2 "2021-06-12T00:36:07Z")

</div>

Not really my ideal solution but I used cephadm to deploy the cluster outside of nomad , that works just wish I could figure out to have nomad control it as well

---

<div class="post-metadata">

**Author:** ![NegativeFeedback](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/negativefeedback/32/18743_2.png) [@NegativeFeedback](https://discuss.hashicorp.com/u/NegativeFeedback)\
**Post date:** [June 13, 2021, 6:37am UTC](https://discuss.hashicorp.com/t/internal-ceph-cluster/25275/3 "2021-06-13T06:37:07Z")

</div>

So I think I should be able to create new volume from nomad using the nomad volume create, but following [Commands: volume create | Nomad by HashiCorp](https://www.nomadproject.io/docs/commands/volume/create) does not see to work.

First of I had to go into the ceph controller and mkdir -p /tmp/csi/keys/ as it could not create a tempt keyfile and now its failing to see monitors?

```auto
$ nomad volume create mysql-volume.hcl
Error creating volume: Unexpected response code: 500 (rpc error: 1 error occurred:
  * controller create volume: CSI.ControllerCreateVolume: volume "mysql_data" snapshot source &{"" ""} is not compatible with these parameters: rpc error: code = InvalidArgument desc = failed to fetch monitor list using clusterID (efddd514-caf9-11eb-a54a-8be5f459e004): missing configuration for cluster ID "efddd514-caf9-11eb-a54a-8be5f459e004"

)

```

```auto
I0613 06:33:09.293467 1 utils.go:166] ID: 11573 Req-ID: mysql_data GRPC request: {"accessibility_requirements":{},"capacity_range":{"limit_bytes":50000000000,"required_bytes":50000000000},"name":"mysql_data","parameters":{"clusterID":"efddd514-caf9-11eb-a54a-8be5f459e004","imageFeatures":"layering","pool":"nomadv"},"secrets":" ***stripped***","volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":1}}]}
E0613 06:33:09.294052 1 rbd_util.go:959] ID: 11573 Req-ID: mysql_data failed getting mons (failed to fetch monitor list using clusterID (efddd514-caf9-11eb-a54a-8be5f459e004): missing configuration for cluster ID "efddd514-caf9-11eb-a54a-8be5f459e004")
E0613 06:33:09.294234 1 utils.go:171] ID: 11573 Req-ID: mysql_data GRPC error: rpc error: code = InvalidArgument desc = failed to fetch monitor list using clusterID (efddd514-caf9-11eb-a54a-8be5f459e004): missing configuration for cluster ID "efddd514-caf9-11eb-a54a-8be5f459e004"

```

```auto
id = "ceph_mysql_data"
name = "mysql_data"
type = "csi"
plugin_id = "ceph-csi"
capacity_max = "50G"
capacity_min = "50G"

capability {
  access_mode = "single-node-writer"
  attachment_mode = "file-system"
}
secrets {
  userID = "myPoolAdmin"
  userKey = "SNIPPED"
}
parameters {
  clusterID = "efddd514-caf9-11eb-a54a-8be5f459e004"
  pool = "nomadv"
  imageFeatures = "layering"
}
mount_options {
  fs_type = "ext4"
}

```

also as a sidenote I get Error decoding the volume definition: At 23:17: unknown slice type: \*ast.LiteralType when the mount options have anything set

---

<div class="post-metadata">

**Author:** ![NegativeFeedback](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/negativefeedback/32/18743_2.png) [@NegativeFeedback](https://discuss.hashicorp.com/u/NegativeFeedback)\
**Post date:** [June 15, 2021, 3:24pm UTC](https://discuss.hashicorp.com/t/internal-ceph-cluster/25275/4 "2021-06-15T15:24:38Z")

</div>

I have not made any progress on this still but very much interested in completing the hashistack life. I dont think there is a better supported CSI implementation but open to suggestion to use other than portworx or ceph if that is what is needed.

Thanks!

---

<div class="post-metadata">

**Author:** ![NegativeFeedback](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/negativefeedback/32/18743_2.png) [@NegativeFeedback](https://discuss.hashicorp.com/u/NegativeFeedback)\
**Post date:** [June 17, 2021, 7:34pm UTC](https://discuss.hashicorp.com/t/internal-ceph-cluster/25275/5 "2021-06-17T19:34:49Z")

</div>

So doing without dynamic volumes, is there a working example of ceph volume hcl registration with out using the create command?

---

<div class="post-metadata">

**Author:** ![lgfa29](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/lgfa29/32/2386_2.png) [@lgfa29](https://discuss.hashicorp.com/u/lgfa29)\
**Post date:** [June 17, 2021, 10:54pm UTC](https://discuss.hashicorp.com/t/internal-ceph-cluster/25275/6 "2021-06-17T22:54:52Z")

</div>

Hi @NegativeFeedback 👋

I don’t know much about Ceph, but have you seen the demo here? Maybe there’s something in there that can help you.

> <https://github.com/hashicorp/nomad/blob/main/demo/csi/ceph-csi-plugin/README.md>

---

<div class="post-metadata">

**Author:** ![NegativeFeedback](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/negativefeedback/32/18743_2.png) [@NegativeFeedback](https://discuss.hashicorp.com/u/NegativeFeedback)\
**Post date:** [June 18, 2021, 4:26pm UTC](https://discuss.hashicorp.com/t/internal-ceph-cluster/25275/7 "2021-06-18T16:26:01Z")

</div>

Hey, thanks for pointing me in that direction! I have seen that and have given that a shot in the past but as I have been learning and adapting my setup to work may have had other variables going, So reset and started from the beginning with that readme without other references and still running into issues.

```auto
$ nomad volume create ./patroni-volume.hcl
Error creating volume: Unexpected response code: 500 (rpc error: 1 error occurred:
        * controller create volume: rpc error: controller create volume: CSI.ControllerCreateVolume: volume "patroni-volume" snapshot source &{"" ""} is not compatible with these parameters: rpc error: code = InvalidArgument desc = failed to fetch monitor list using clusterID (efddd514-caf9-11eb-a54a-8be5f459e004): missing configuration for cluster ID "efddd514-caf9-11eb-a54a-8be5f459e004"

```