I have a use case where I need to mount a usb serial device into a docker container that has been deployed using Nomad and upon looking through the documentation there only seems to be the Nvidia device plugin available.
Because of this I have started to learn how to make a device plugin and have started off with a raw USB device which will just mount the “/dev/bus/usb/{bus}/{address}” into the container given that you have provided a vendor and product id. Bare in mind I am new to golang, new to Nomad, and also new to the dev/udev system in linux so it’s been quite fun to mess around with it, however, because of that I am not sure if I have built the plugin in a way that is useful to anyone other than me .
I originally started with the skeleton repo that HashiCorp provided and also found another user who did a HashiTalk demo which had a base setup for at least finding a USB device so a lot of what I have has come from both of these repositories.
My main reason for this post is to see if anyone else has a use for these kind of device plugins, as I plan on making another for mounting a serial device (e.g. /dev/ttyACM0) into a deployed task, and if so, whether I should see about making them more reasonable/generic to use for others before deciding to add it to the community plugin list. This is also a check to see if anyone has already made these plugins and has not put them onto the plugin list yet as they would probably be of a better quality than what I have done so far
My second reason for this post is to see if anyone has any pointers on how to make a good device plugin as i’m still not super confident in what i’ve done so far and there are not many examples that I have seen so far (except the Nvidia one).
As a follow up (mainly for pointers) I have added what I have done so far to a repo.
I have a working plugin which is currently built for linux/armv7, linux/arm64, linux/amd64, and linux/i368.
The plugin allows the agent to include/exclude specific vendor ids from fingerprinting and also allows the device stanza to define what vendor/product id it wants
This is really cool, @CarbonCollins! I know there are folks running Nomad homelabs on RaspPi clusters who might benefit from being able to mount external USB storage devices. I’d love to get a PR for adding this to the [Community Device Driver](Device Plugins: Community Supported | Nomad by HashiCorp] docs. Having an example that it can be done from someone in the community can serve as inspiration for someone else to do it too!
I’m one of those folks who are currently in the process of moving their current homelab over to using Nomad which is a mix of amd64 machines and armv7 RasPis so I would defiantly find this kind of thing useful! I’m personally needing this kind of functionality to mount a USB ZigBee gateway into a deployed task.
The few days that I have been experimenting with this I’ve run into several questions and issues which I am not sure I can really answer yet. Some questions include:
What vendor and Model should I expose to nomad, should it be the VID:PID or should it be the registered name of the USB device?
How do I specify what to mount into the container and how will it handle changes.
What attributes should I pull from the device and make available to Nomad and in what format should they be
How should newer versions of the plugin be deployed to clients (ssh-ing into each client is getting quite tedious while i’m experimenting and not on my dev machine)
I plan on experimenting with it more tonight and tomorrow evening so we shall see if I make any progress on any of the questions I have and with the current state of the driver being not very stable I am a little hesitant to open the PR to the community page just yet.
So I have reached a point with this plugin where it is a lot more stable (it has at least been running on my cluster for a least a week now without any issues)
Some things I have learned / discovered is that I am not able to easily discover the dev path in which the USB device is associated with and thus I am not able to easily provide the dev path for mounting into containerised workloads (e.g. docker) For now the task I have is manually mounting /dev/ttyACM0 on a node that is known to have the USB connected too but this is not a very sustainable way of doing it .
My meaning of easily in this is that I cant find a maintained golang module which provides this functionality and I don’t know enough about this topic to be confident enough to even create one for use! (yet at least…)
If anyone has any pointers on how I could get the dev path for a connected USB I would be very thankful!
I have however at least answered some of my previous questions and have settled on using the VID and PID for the product and model values so that task definitions can be used like:
device "c1f0/usb/0030" {}
and in regards to the attributes I have plenty to choose from so I am still working out which ones would be reasonable to expose to nomad.
I’m now assuming that the intention with updating device drivers is that someone has to manually go in and update the plugin on the nodes as there is no built in mechanism for doing this (that I know of)
As a final note i’m also currently looking into adding at least the current version to the community device drivers page under a beta flag or something of the like until I have fleshed it out a bit more or until I find a good way of doing it which will work properly with containerised workloads.
Now I can see the USB device with „lsusb“ inside the container, but on access it will just throw an „Could not open“ libusb error from the application (hmland).
You mentioned the manually mounting of the USB device on a node - did i missed something here?
Or could it rely on limitations of the podman plugin?
I’ve not tried with podman unfortunatly, only with docker.
I can see about giving podman a go on my stack to see if I get the same issue though
in regards to the manual mounting, I have been mounting related dev nodes like /dev/ttyUSB0 into the containers instead of the bus dev mount /dev/bus/usb/001/001 as I’ve had a few issues with some of my USB devices reconnecting and getting a new bus number and thus new dev path after the workload has started, so I will also give that one a go in docker and podman to see if I get a similar error to what you are getting.
It may take a little time for me to look into it so if you find anything out in the meantime please let me know
Yes mounting the bus dev will not be very reliable - used it just for testing.
But if I understand it correctly your usb-device-plugin will take care of mouting the device into the container, right?
I would like it to be able to do that yes, but I’m currently struggling to find a reasonable way of discovering what devices are related to that usb device, the google/gousb golang package only makes use of libusb which does not allow me to get this information. I’m currently on the look out for a currently maintained golang package that can provide this functionality, and if I cannot find one, ill look into writing my own
.
.
