New Guides for Nomad beta v0.11.0!

We have created a couple new guides to for Nomad beta v0.11.0 features.

In the Express Inter-job Dependencies with Init Tasks guide, you’ll use Nomad task dependencies to create a job that will wait for a service advertised in Consul to become available.

In the Stateful Workloads with Container Storage Interface guide, you’ll deploy the AWS EBS Container Storage Interface (CSI) plugin and MySQL workload that uses a registered EBS volume for persistent storage.

Download Nomad beta v0.11.0 now to try out the guides!


Any example for storage in a dev environment or in digitalocean?

Any chance of changing/updating the title of the OP? So as to let the reader know that it would work with the latest versions as well ?!

The “CSI driver for DigitalOcean” is geared toward Kubernetes, not for Nomad. Played with it for several hours today and finally gave up.

HI @SunSparc Thanks for using Nomad!

I’m sorry to hear you have been having issues with the DO experience. I am happy to say that Nomad 1.3.0 is in RC1 right now and has numerous improvements to the CSI story. Were you using that for your testing? If not, maybe give that a shot and see if things go better.

I will also relay your experience to our team and see if the guides need any updates. If you can provide any specific information about what didn’t go well it would be really helpful as we consider what might need updating.

Thanks again for using Nomad and thank you for taking time to let us know where we can do better.

Thanks for the update. I was using Nomad version 1.2.6. I will give 1.3.0 a try and see how it goes.

Looking through the “CSI driver for DigitalOcean” (linked to by @fhemberger) repository, the only mention of “Nomad” in the entire repository is in an issue that was opened back in 2020.

The last comment of that issue directs people to the DigitalOcean CSI Demo in the Nomad repository.

The demo is Terraform centric. Is it possible to get CSI working in Nomad without the use of Terraform? The digitalocean_volume resource probably can only be done with Terraform, but the rest seems like it could be done in Nomad, though there are data points that need to be included for them to work… :thinking:

For me, the process of getting CSI working in Nomad has been rough. There are so many nuances that make the learning curve extremely steep. Such as:

  • if using a templatefile in the jobspec, how can I refer to a file that is in the same directory with the *.tf files that are referencing it
  • what exactly is the plugin_id that is referred to in the resource nomad_volume.test_volume.plugin_id
  • why does "--endpoint=${CSI_ENDPOINT}" fail if including the jobspec as a heredoc (needs $$)
  • (many other fiddly bits :slight_smile: )

I was able to get a volume created at DigitalOcean. I was able to get the do-csi-plugin job running in Nomad. I was able to create a nomad_volume in Nomad that connected to the DigitalOcean volume. Then I started up a custom job with a volume stanza of type csi and the source being the volume I setup in previous step. The service failed with this message:

failed to setup alloc: pre-run hook "csi_hook" failed:
  node plugin returned an internal error, check the plugin allocation logs for more information:
  rpc error: code = Internal desc = mounting failed:
    exit status 1 cmd: 'mount -o bind /dev/sda /local/csi/per-alloc/3704f0d8-71d4-b0fe-16f1-8be905a5a17f/loki-persistent-storage-volume/rw-block-device-single-node-writer'
  output: "mount: permission denied (are you root?)\n"

Upon looking in the CSI plugin logs I found this:

msg="method failed"
error="rpc error:
         code = FailedPrecondition
         desc = volume \"27127649-d0a2-13ec-813a-0a58ac1428a4\" is attached to the wrong droplet (211207561), detach the volume to fix it"

Apparently I need to make sure that the job runs on the same node that the volume gets attached to. I need to look into that further, but I wanted to leave an update here for the moment.