# Intermittent I/O Errors While Using NFS CSI plugin **URL:** \ **Category:** Nomad\ **Created:** [February 12, 2024, 6:35pm UTC](https://discuss.hashicorp.com/t/intermittent-i-o-errors-while-using-nfs-csi-plugin/62704 "2024-02-12T18:35:05Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![nopersonalspace](https://avatars.discourse-cdn.com/v4/letter/n/ecae2f/32.png) [@nopersonalspace](https://discuss.hashicorp.com/u/nopersonalspace)\ **Post date:** [February 12, 2024, 6:35pm UTC](https://discuss.hashicorp.com/t/intermittent-i-o-errors-while-using-nfs-csi-plugin/62704/1 "2024-02-12T18:35:05Z")
I’ve been trying to run a nomad cluster with a NAS as the storage backend for it (meaning no local volumes on any of the servers/nodes) and I’m currently using [the nfs csi driver](https://github.com/kubernetes-csi/csi-driver-nfs) but I’ve been having issues. I’ve been seeing jobs emitting I/O errors like `IOError: SQLitePCL.pretty.SQLiteException: disk I/O error` intermittently, and it seems like it has something to do with the nfs csi plugin. When I re-start the allocation it always goes away no problem. Has anyone else using the nfs csi plugin had issues like this? Is there another way to use nfs, or should I abandon it and try smb (I’ve always had better luck with SMB in the past, honestly) or something else? ---
**Author:** ![nopersonalspace](https://avatars.discourse-cdn.com/v4/letter/n/ecae2f/32.png) [@nopersonalspace](https://discuss.hashicorp.com/u/nopersonalspace)\ **Post date:** [February 16, 2024, 4:49pm UTC](https://discuss.hashicorp.com/t/intermittent-i-o-errors-while-using-nfs-csi-plugin/62704/2 "2024-02-16T16:49:26Z")
I’m seeing this error sometimes, which I’m assuming is related: `stale NFS file handle` ---
**Author:** ![shaun.coss](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/shaun.coss/32/16400_2.png) [@shaun.coss](https://discuss.hashicorp.com/u/shaun.coss)\ **Post date:** [February 16, 2024, 5:04pm UTC](https://discuss.hashicorp.com/t/intermittent-i-o-errors-while-using-nfs-csi-plugin/62704/3 "2024-02-16T17:04:11Z")
I would avoid using that particular driver. I would recommend the Democratic CSI driver. ---
**Author:** ![nopersonalspace](https://avatars.discourse-cdn.com/v4/letter/n/ecae2f/32.png) [@nopersonalspace](https://discuss.hashicorp.com/u/nopersonalspace)\ **Post date:** [February 16, 2024, 5:10pm UTC](https://discuss.hashicorp.com/t/intermittent-i-o-errors-while-using-nfs-csi-plugin/62704/4 "2024-02-16T17:10:13Z")
Okay, thanks. Yeah it seems like the driver is faulty. I’ll switch over to Democratic CSI and see if that helps. ---
**Author:** ![nopersonalspace](https://avatars.discourse-cdn.com/v4/letter/n/ecae2f/32.png) [@nopersonalspace](https://discuss.hashicorp.com/u/nopersonalspace)\ **Post date:** [February 27, 2024, 9:09pm UTC](https://discuss.hashicorp.com/t/intermittent-i-o-errors-while-using-nfs-csi-plugin/62704/6 "2024-02-27T21:09:18Z")
Does anyone use democratic-csi for nfs and have an example config to share? I can’t get this working and the documentation is pretty obtuse… ---
**Author:** ![benvanstaveren](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/benvanstaveren/32/1064_2.png) [@benvanstaveren](https://discuss.hashicorp.com/u/benvanstaveren)\ **Post date:** [March 12, 2024, 12:05pm UTC](https://discuss.hashicorp.com/t/intermittent-i-o-errors-while-using-nfs-csi-plugin/62704/8 "2024-03-12T12:05:35Z")
It’s not necessarily an issue with the CSI driver, after all the underlying system is NFS so you may have issues with connectivity to your NAS. ---
**Author:** ![nopersonalspace](https://avatars.discourse-cdn.com/v4/letter/n/ecae2f/32.png) [@nopersonalspace](https://discuss.hashicorp.com/u/nopersonalspace)\ **Post date:** [March 26, 2024, 7:39pm UTC](https://discuss.hashicorp.com/t/intermittent-i-o-errors-while-using-nfs-csi-plugin/62704/9 "2024-03-26T19:39:20Z")
I guess it’s possible, but I don’t know what those issues would be. I have the NAS connected to various other machines (via SMB) and have not noticed any connectivity issues, although none of those machines are needing the same sort of persistent connection that Nomad does. None of the other machines use NFS though, so like I mentioned earlier maybe I can just try SMB. I was trying to write a template that would work for SMB but I was having trouble getting it to work. ---
**Author:** ![nopersonalspace](https://avatars.discourse-cdn.com/v4/letter/n/ecae2f/32.png) [@nopersonalspace](https://discuss.hashicorp.com/u/nopersonalspace)\ **Post date:** [March 26, 2024, 7:45pm UTC](https://discuss.hashicorp.com/t/intermittent-i-o-errors-while-using-nfs-csi-plugin/62704/10 "2024-03-26T19:45:47Z")
I’m using Nix to configure the machines in the cluster running nomad, so another option I’ve considered is that I could just mount the SMB/NFS shares to the host machine and then use docker volume mounts to access them that way. Is that advisable? Would I run into issues with all of the containers trying to access the same SMB/NFS filesystem at once? ---
**Author:** ![benvanstaveren](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/benvanstaveren/32/1064_2.png) [@benvanstaveren](https://discuss.hashicorp.com/u/benvanstaveren)\ **Post date:** [March 26, 2024, 8:04pm UTC](https://discuss.hashicorp.com/t/intermittent-i-o-errors-while-using-nfs-csi-plugin/62704/11 "2024-03-26T20:04:23Z")
> [@nopersonalspace](#): > > I’m using Nix to configure the machines in the cluster running nomad, so another option I’ve considered is that I could just mount the SMB/NFS shares to the host machine and then use docker volume mounts to access them that way. The better way to do that is to mount them to the node(s), and set up host volumes - that way if you ever switch back from host volume to CSI it’s easy, and it works for non-docker drivers too. And it won’t be a problem with access, one of our legacy bits at `$work` uses CSI NFS (with the occasional same issues, hence my comment) and we have about 60 apps across 9 nodes reading/writing. We’ve never had write or read contention, only outright “the NFS isn’t responding” type errors. ---
**Author:** ![nopersonalspace](https://avatars.discourse-cdn.com/v4/letter/n/ecae2f/32.png) [@nopersonalspace](https://discuss.hashicorp.com/u/nopersonalspace)\ **Post date:** [March 26, 2024, 8:18pm UTC](https://discuss.hashicorp.com/t/intermittent-i-o-errors-while-using-nfs-csi-plugin/62704/12 "2024-03-26T20:18:27Z")
Interesting, by host volumes you mean [this](https://developer.hashicorp.com/nomad/docs/configuration/client#host_volume-stanza)? That seems like it could work, but do you by chance know if there is a way to specify a sub-directory when using these in a job? I’d rather not have to make a separate `host_volume` for each job because I have to re-boot the node to apply changes to the client config each time. ---
**Author:** ![benvanstaveren](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/benvanstaveren/32/1064_2.png) [@benvanstaveren](https://discuss.hashicorp.com/u/benvanstaveren)\ **Post date:** [March 26, 2024, 8:35pm UTC](https://discuss.hashicorp.com/t/intermittent-i-o-errors-while-using-nfs-csi-plugin/62704/13 "2024-03-26T20:35:38Z")
Yep, those 🙂 As far as I know you can reload Nomad and it’ll pick them up as well so no full restart required on the client node. You can’t directly mount a subdirectory, but you can do something (which we have done) like this: ```auto host_volume "job-storage" { path = "/mounted/over/nfs" read_only = false } ``` Then in a job you declare and mount: ```auto volume "storage" { type = "host" read_only = false source = "job-storage" } ... ... ... volume_mount { volume = "storage" destination = "/storage" propagation_mode = "host-to-task" } ``` And your jobs then need to create their own appropriate subdirectory under /storage - assuming, of course, there is some form of unique identifier for a single run of the job (ID from a database, parameters, or something similar like that). Since host volumes (and NFS/SMB) have absolutely no problem with multiple readers/writers, it should “just work™” - but YMMV on that 🙂 Alternatively go with the direct docker volume mount and skip the host volumes. The reason I suggested host volumes is more to make the underlying mechanism opaque to the job so it becomes eas(y|ier) to change in the future if need be.