Long time Nomad user here. So far, we’ve been running Nomad mostly via host mounts (backed by glusterfs under the hood). Would love to migrate to CSI to be more flexible, but I am not sure which CSI plugins are working properly with Nomad.
Anything you can recommend? Ideally something we could self-host. Thanks for your insights!
Hi @shochdoerfer,
Long time Nomad user here.
Great to hear, hope things have been working well for you.
which CSI plugins are working properly with Nomad
In theory, any CSI plugin which correctly implements the specification should work properly with Nomad. Nomad does have some open issues to support initial specification details and I would suggest looking through this GitHub filter from time to time.
Anything you can recommend? Ideally something we could self-host.
The Nomad engineering team runs AWS EBS and EFS within our E2E suite, however, this is obviously not self-hosted. I’d therefore love to hear and encourage the community to discuss what CSI implementations they are running.
The demo/csi directory has some examples of CSI plugins which may be useful during any proof of concept you perform. Tim (Nomad Engineer and CSI SME) wrote these CSI plugin nomad-packs which might also be useful.
Thanks,
jrasell and the Nomad team
Thanks a lot! I know in theory all CSI plugins should be compatible. I checked a few CSI implementations on GitHub and found issues here and there mentioning problems with Nomad. That’s why I thought, I better ask here instead of testing a bunch of solutions and just wasting time.
Will check the demo/csi link. Thanks!
I’ve got a setup with Ceph CSI running in a small Homelab setup and haven’t had any problems with it. Job stop/start as well as job migration from one node to another have been working without issue. And you can self-host Ceph (albeit there are definitely simpler storage solutions with CSI support).
hi,
I think, that is one of the main issues with Nomad and CSI: “in theory”. We have also a Ceph working as storage for our Proxmox, but I would not use it for Nomad in our case. I searched a lot for users, who tried for example using Minio with S3 as backend … but no success. In most cases, the most common used - shared - filesystem … is still NFS.
In the end, we decided to use SeaweedFS, which has worked quite well with Nomad so far.
At work, in a development environment, I currently use the CSI driver from JuiceFS backed by a Minio cluster.
This allows us to deploy with CI/CD jobs our cutting-edge version of developments while keeping data persistent.
At this time we run 8 jobs running 20 allocations on 5 small nodes (8Gb RAM and 4 vCPU).
Thx for sharing. How’s the performance of JuiceFS and Minio? In general, we are happy with our SeaweedFS setup, but we have a few jobs that we had to move away, like GitLab, which became super slow with the latest upgrade. Also, some other tools had problems writing files to the SeaweedFS volume.
Well, in my case, everything,from Nomad agent to Minio nodes, runs as VMs inside a VSphere cluster using old hardware so performances are enough for development platform but I guess it would be better if the Minio cluster was on bare metal machines with good disks.
I’ve never performed any performance tests as this is only for the developers team. Moreover, I’m only the software architect and not working as system engineer 
On the configuration side, it was almost super easy. Minio cluster was provisioned by Terraform and Puppet and the JuiceFS CSI tutorial is good enough to start using it in few minutes.
The little plus is that we needed some S3 endpoints for applications running inside the Nomad cluster so integrating Minio as a Consul terminating gateway did the trick very easily.
While doing another search, I found this.
This would eliminate the need for JuiceFS.
OTOH, JuiceFS allows to use other storage backends and acts as a middleware and thus allows to change it transparently from Nomad point of view
Nice, will have a look. That would reduce the complexity even more.
Can you explain why you mean by “some tools had problems writing files to SeaweedFS” ?
Do you know now why performance degraded with the last upgrade ?
Maybe SeaweedFS could be a parallel solution if our infrastructure goes to production.
In general, SeaweedFS works fine for our workload. We have a lot of jobs running and have no issues with the performance at all.
It’s just GitLab that was a bit slow after we migrated it to our SeaweedFS cluster. One of the recent GitLab updates made it even slower. It was barely usable for my team. GitLab does a lot of Git operations, obviously, and thus needs a fast disk. I didn’t manage to debug the “exact” issue why it got so slow or what the reason was.
I am still a big fan of SeaweedFS, it was easy to set up (single binary / Docker image) and is super flexible regarding data replication.