Hi
I’m having some issues with CSI volumes in which I cant determine the root cause to be able to raise bug reports in GitHub 
I’m not sure if it is just one issue or many, or if its even Nomad causing the issues and instead the CSI plugin I am using.
I’m running a single cluster at home which operates 20-30 services, some of which are stateful, across 7 nodes and 3 servers of varying architectures and base OS’s. I originally used only host volumes but wanted to make it more dynamic by using CSI Volumes with my SMB NAS. For this I am using the Democratic CSI - smb-client plugin for mounting volumes. The volumes themselves work for the most part but I keep on running into issues with allocations.
I have jobs defined for the CSI Nodes and Controllers and volumes registered using the following:
Controller
job "storage-plugin-axion-controller" {
type = "service"
region = "global"
datacenters = ["proxima"]
priority = 80
group "controller" {
count = 3
constraint {
distinct_hosts = true
}
task "plugin" {
driver = "docker"
config {
image = "[[ .democraticImage ]]"
args = [
"--csi-mode=controller",
"--csi-version=[[ .csiVersion ]]",
"--csi-name=org.democratic-csi.smb-client",
"--driver-config-file=${NOMAD_TASK_DIR}/driver-config-file.yaml",
"--server-socket=/csi-data/csi.sock"
]
}
vault {
policies = ["cluster-storage"]
}
csi_plugin {
id = "axion-proxima"
type = "controller"
mount_dir = "/csi-data"
}
template {
data = <<EOH
{{ with secret "cluster-storage/proxima" }}
driver: smb-client
smb:
shareHost: {{ index .Data.data "AXION_HOST" }}
shareBasePath: "{{ index .Data.data "AXION_SHARE" }}"
controllerBasePath: "/storage"
dirPermissionsMode: "0777"
dirPermissionsUser: 1005
dirPermissionsGroup: 1005
{{ end }}
EOH
destination = "${NOMAD_TASK_DIR}/driver-config-file.yaml"
}
resources {
cpu = 30
memory = 75
}
}
}
}
Node
job "storage-plugin-axion-node" {
type = "system"
region = "global"
datacenters = ["proxima"]
priority = 79
constraint {
operator = "distinct_hosts"
value = true
}
group "node" {
task "plugin" {
driver = "docker"
config {
image = "[[ .democraticImage ]]"
args = [
"--csi-mode=node",
"--csi-version=[[ .csiVersion ]]",
"--csi-name=org.democratic-csi.smb-client",
"--driver-config-file=${NOMAD_TASK_DIR}/driver-config-file.yaml",
"--server-socket=/csi-data/csi.sock"
]
privileged = true
}
vault {
policies = ["cluster-storage"]
}
csi_plugin {
id = "axion-proxima"
type = "node"
mount_dir = "/csi-data"
}
template {
data = <<EOH
{{ with secret "cluster-storage/proxima" }}
driver: smb-client
smb:
shareHost: {{ index .Data.data "AXION_HOST" }}
shareBasePath: "{{ index .Data.data "AXION_SHARE" }}"
controllerBasePath: "/storage"
dirPermissionsMode: "0777"
dirPermissionsUser: 1005
dirPermissionsGroup: 1005
{{ end }}
EOH
destination = "${NOMAD_TASK_DIR}/driver-config-file.yaml"
}
resources {
cpu = 30
memory = 75
}
}
}
}
Volume
id = "[[ .volumeName ]]"
name = "[[ .volumeName ]]"
type = "csi"
external_id = "[[ .volumeName ]]"
plugin_id = "axion-proxima"
capacity_max = "2G"
capacity_min = "1G"
capability {
access_mode = "single-node-reader-only"
attachment_mode = "file-system"
}
capability {
access_mode = "single-node-writer"
attachment_mode = "file-system"
}
mount_options {
fs_type = "cifs"
mount_flags = [
"username=[[ .volumeUser ]]",
"password=[[ .volumePass ]]",
"vers=3",
"uid=[[ .defaultUserId ]]",
"gid=[[ .defaultGroupId ]]",
"nolock"
]
}
secrets {
username = "[[ .volumeUser ]]"
password = "[[ .volumePass ]]"
}
context {
node_attach_driver = "smb"
provisioner_driver = "smb-driver"
server = "[[ .axionServer ]]"
share = "[[ .axionShare ]]"
}
One of the issues I run into is “phantom” allocations that remain after a job either finishes or restarts. The job itself is marked as completed but the CSI volume remains allocated. This then prevents the job from starting again as the volume is already allocated. I can’t clear this allocation without manually running nomad volume detach {volumeName} {nodeId} or in some cases I have to deregister and re-create the volume fully. The UI in different views seem to disagree on if there is an allocation or not, the Volumes Summary page will show a number of #Allocs but going to the Volume Details page will show No Read/Write Allocations for that volume. The Volumes themselves will also show that they are Node Only or 3/0 for the Controller Health which leads me onto the next issue.
The second issue is that the CSI Controller do not seem to register themselves with Nomad correctly or reliably which leads to the CSI Plugin to report that its either Node Only, 3/0, or 2/3 for the Controller Health. I’m also not sure if this is the cause of the first issue or if its something separate . The controller job itself reports that all instances are running and they all report the Plugin became healthy when looking at the Recent Events. I can see that the controller is at least responding to requests as the logs report new response and new request info logs, however, I am not very versed in understanding what it is doing as this is my first entry into the world of CSI . I do occasionally see an error in the logs which looks like:
error: failed to create csi client: failed to open grpc connection to addr: /var/lib/nomad/client/csi/node/axion-proxima/csi.sock, err: context deadline exceeded
The third issue normally presents itself when trying to detach a volume to work around the first issue. I ocasionaly get a 500 error when running the nomad volume detach command which returns something like:
Volume Detach Error
Error detaching volume: Unexpected response code: 500 (rpc error: 1 error occurred:
* could not detach from node: node detach volume: CSI.NodeDetachVolume: rpc error: code = Aborted desc = operation locked due to in progress operation(s): ["volume_id_deconz-data"]
I’m not sure if this issues appeared because of either of the previous issues or if its something else but once it occurs it sometimes clears itself after waiting 5-10 mins but other times it never resolves itself.
Overall I am not sure if its just the way I am using the CSI Volumes that is causing these issues, if its a config error, or something rising because its still a beta feature. I’m also not sure what I should raise to GitHub as an issue as I don’t really know the cause of these issues. I am already watching a few GitHub issues which seem related/similar to what I am having such as:
#10052 Referential integrity issue with CSI volumes.
#10833 CSI node unpublish/unstage should avoid requiring server
#10927 Constraint “CSI volume has exhausted its available writer claims”: 1 nodes excluded by filter ← seems to be very similar to the first issue at least
I also raised a previous issue where I mentioned some of the above:
#11174 Nomad server panic with SIGSEGV signal when using CSI volumes.
Minus these issues I’m really happy that I can have stateful data which can be mounted across all nodes in my cluster, it has been quite helpful when I had a node die on me 
(SD card failure in a Raspberry Pi)