In the Stateful Workloads with Container Storage Interface guide, you’ll deploy the AWS EBS Container Storage Interface (CSI) plugin and MySQL workload that uses a registered EBS volume for persistent storage.
Download Nomad beta v0.11.0 now to try out the guides!
I’m sorry to hear you have been having issues with the DO experience. I am happy to say that Nomad 1.3.0 is in RC1 right now and has numerous improvements to the CSI story. Were you using that for your testing? If not, maybe give that a shot and see if things go better.
I will also relay your experience to our team and see if the guides need any updates. If you can provide any specific information about what didn’t go well it would be really helpful as we consider what might need updating.
Thanks again for using Nomad and thank you for taking time to let us know where we can do better.
The last comment of that issue directs people to the DigitalOcean CSI Demo in the Nomad repository.
The demo is Terraform centric. Is it possible to get CSI working in Nomad without the use of Terraform? The digitalocean_volume resource probably can only be done with Terraform, but the rest seems like it could be done in Nomad, though there are data points that need to be included for them to work…
For me, the process of getting CSI working in Nomad has been rough. There are so many nuances that make the learning curve extremely steep. Such as:
if using a templatefile in the jobspec, how can I refer to a file that is in the same directory with the *.tf files that are referencing it
what exactly is the plugin_id that is referred to in the resource nomad_volume.test_volume.plugin_id
why does "--endpoint=${CSI_ENDPOINT}" fail if including the jobspec as a heredoc (needs $$)
(many other fiddly bits )
I was able to get a volume created at DigitalOcean. I was able to get the do-csi-plugin job running in Nomad. I was able to create a nomad_volume in Nomad that connected to the DigitalOcean volume. Then I started up a custom job with a volume stanza of type csi and the source being the volume I setup in previous step. The service failed with this message:
failed to setup alloc: pre-run hook "csi_hook" failed:
node plugin returned an internal error, check the plugin allocation logs for more information:
rpc error: code = Internal desc = mounting failed:
exit status 1 cmd: 'mount -o bind /dev/sda /local/csi/per-alloc/3704f0d8-71d4-b0fe-16f1-8be905a5a17f/loki-persistent-storage-volume/rw-block-device-single-node-writer'
output: "mount: permission denied (are you root?)\n"
Upon looking in the CSI plugin logs I found this:
level=error
msg="method failed"
error="rpc error:
code = FailedPrecondition
desc = volume \"27127649-d0a2-13ec-813a-0a58ac1428a4\" is attached to the wrong droplet (211207561), detach the volume to fix it"
host_id=211207561
method=/csi.v1.Controller/ControllerPublishVolume
region=nyc3
version=v4.0.0
Apparently I need to make sure that the job runs on the same node that the volume gets attached to. I need to look into that further, but I wanted to leave an update here for the moment.
)