Long time Nomad user here. So far, we’ve been running Nomad mostly via host mounts (backed by glusterfs under the hood). Would love to migrate to CSI to be more flexible, but I am not sure which CSI plugins are working properly with Nomad.
Anything you can recommend? Ideally something we could self-host. Thanks for your insights!
Hi @shochdoerfer,
Long time Nomad user here.
Great to hear, hope things have been working well for you.
which CSI plugins are working properly with Nomad
In theory, any CSI plugin which correctly implements the specification should work properly with Nomad. Nomad does have some open issues to support initial specification details and I would suggest looking through this GitHub filter from time to time.
Anything you can recommend? Ideally something we could self-host.
The Nomad engineering team runs AWS EBS and EFS within our E2E suite, however, this is obviously not self-hosted. I’d therefore love to hear and encourage the community to discuss what CSI implementations they are running.
The demo/csi directory has some examples of CSI plugins which may be useful during any proof of concept you perform. Tim (Nomad Engineer and CSI SME) wrote these CSI plugin nomad-packs which might also be useful.
Thanks,
jrasell and the Nomad team
Thanks a lot! I know in theory all CSI plugins should be compatible. I checked a few CSI implementations on GitHub and found issues here and there mentioning problems with Nomad. That’s why I thought, I better ask here instead of testing a bunch of solutions and just wasting time.
Will check the demo/csi link. Thanks!
I’ve got a setup with Ceph CSI running in a small Homelab setup and haven’t had any problems with it. Job stop/start as well as job migration from one node to another have been working without issue. And you can self-host Ceph (albeit there are definitely simpler storage solutions with CSI support).
hi,
I think, that is one of the main issues with Nomad and CSI: “in theory”. We have also a Ceph working as storage for our Proxmox, but I would not use it for Nomad in our case. I searched a lot for users, who tried for example using Minio with S3 as backend … but no success. In most cases, the most common used - shared - filesystem … is still NFS.
In the end, we decided to use SeaweedFS, which has worked quite well with Nomad so far.