USB/Serial Device plugins

Nomad
#1

Hi,

I have a use case where I need to mount a usb serial device into a docker container that has been deployed using Nomad and upon looking through the documentation there only seems to be the Nvidia device plugin available.

Because of this I have started to learn how to make a device plugin and have started off with a raw USB device which will just mount the “/dev/bus/usb/{bus}/{address}” into the container given that you have provided a vendor and product id. Bare in mind I am new to golang, new to Nomad, and also new to the dev/udev system in linux so it’s been quite fun to mess around with it, however, because of that I am not sure if I have built the plugin in a way that is useful to anyone other than me :slightly_smiling_face:.

I originally started with the skeleton repo that HashiCorp provided and also found another user who did a HashiTalk demo which had a base setup for at least finding a USB device so a lot of what I have has come from both of these repositories.

My main reason for this post is to see if anyone else has a use for these kind of device plugins, as I plan on making another for mounting a serial device (e.g. /dev/ttyACM0) into a deployed task, and if so, whether I should see about making them more reasonable/generic to use for others before deciding to add it to the community plugin list. This is also a check to see if anyone has already made these plugins and has not put them onto the plugin list yet as they would probably be of a better quality than what I have done so far :stuck_out_tongue:

My second reason for this post is to see if anyone has any pointers on how to make a good device plugin as i’m still not super confident in what i’ve done so far and there are not many examples that I have seen so far (except the Nvidia one).

1 Like
#2

As a follow up (mainly for pointers) I have added what I have done so far to a repo.

I have a working plugin which is currently built for linux/armv7, linux/arm64, linux/amd64, and linux/i368.
The plugin allows the agent to include/exclude specific vendor ids from fingerprinting and also allows the device stanza to define what vendor/product id it wants

The plugin can be found at: Steven Collins / Nomad USB Device Plugin · GitLab for viewing or giving it a go.

edit: Removed statement about not being able to filter on attributes as it now can
edit2: Added arm64/i386 architectures to list

1 Like
#3

This is really cool, @CarbonCollins! I know there are folks running Nomad homelabs on RaspPi clusters who might benefit from being able to mount external USB storage devices. I’d love to get a PR for adding this to the [Community Device Driver](Device Plugins: Community Supported | Nomad by HashiCorp] docs. Having an example that it can be done from someone in the community can serve as inspiration for someone else to do it too!

#4

I’m one of those folks who are currently in the process of moving their current homelab over to using Nomad which is a mix of amd64 machines and armv7 RasPis so I would defiantly find this kind of thing useful! I’m personally needing this kind of functionality to mount a USB ZigBee gateway into a deployed task.

The few days that I have been experimenting with this I’ve run into several questions and issues which I am not sure I can really answer yet. Some questions include:

  • What vendor and Model should I expose to nomad, should it be the VID:PID or should it be the registered name of the USB device?
  • How do I specify what to mount into the container and how will it handle changes.
  • What attributes should I pull from the device and make available to Nomad and in what format should they be
  • How should newer versions of the plugin be deployed to clients (ssh-ing into each client is getting quite tedious while i’m experimenting and not on my dev machine)

I plan on experimenting with it more tonight and tomorrow evening so we shall see if I make any progress on any of the questions I have and with the current state of the driver being not very stable I am a little hesitant to open the PR to the community page just yet.

edit: spelling

#5

So I have reached a point with this plugin where it is a lot more stable (it has at least been running on my cluster for a least a week now without any issues)

Some things I have learned / discovered is that I am not able to easily discover the dev path in which the USB device is associated with and thus I am not able to easily provide the dev path for mounting into containerised workloads (e.g. docker) For now the task I have is manually mounting /dev/ttyACM0 on a node that is known to have the USB connected too but this is not a very sustainable way of doing it :smiley: .

My meaning of easily in this is that I cant find a maintained golang module which provides this functionality and I don’t know enough about this topic to be confident enough to even create one for use! (yet at least…)

If anyone has any pointers on how I could get the dev path for a connected USB I would be very thankful! :smiley:

I have however at least answered some of my previous questions and have settled on using the VID and PID for the product and model values so that task definitions can be used like:

device "c1f0/usb/0030" {}

and in regards to the attributes I have plenty to choose from so I am still working out which ones would be reasonable to expose to nomad.

I’m now assuming that the intention with updating device drivers is that someone has to manually go in and update the plugin on the nodes as there is no built in mechanism for doing this (that I know of)

As a final note i’m also currently looking into adding at least the current version to the community device drivers page under a beta flag or something of the like until I have fleshed it out a bit more or until I find a good way of doing it which will work properly with containerised workloads.