Cannot utilize EBS volume

Nomad
1

We are receiving the following error when a job TRIES to mount a volume provided by CSI_PLUGIN

failed to setup alloc: pre-run hook "csi_hook" failed: claim volumes: rpc error: controller publish: attach volume: controller attach volume: volume "vol-01c4001a2ee27a75a" is already published on another node and does not have MULTI_NODE volume capability: rpc error: code = FailedPrecondition desc = i-0ad6ea5f5ff9cc515

Any ideas how to attack this problem?

nomad v 1.0.4
csi plugin 0.10.1

2

I fixed that problem by removing the CSI directory under opt/nomad/client

but now i am facing the following error:

failed to setup alloc: pre-run hook "csi_hook" failed: rpc error: code = InvalidArgument desc = Device path not provided

logs

I0426 21:53:11.440300       1 node.go:498] NodeGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0426 21:53:41.441913       1 node.go:498] NodeGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0426 21:54:11.442721       1 node.go:498] NodeGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0426 21:54:41.443488       1 node.go:498] NodeGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0426 21:55:11.444591       1 node.go:498] NodeGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0426 21:55:41.445542       1 node.go:498] NodeGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0426 21:56:11.446301       1 node.go:498] NodeGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0426 21:56:41.446972       1 node.go:498] NodeGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0426 21:56:54.554547       1 node.go:105] NodeStageVolume: called with args {VolumeId:vol-0998a88f677e69072 PublishContext:map[] StagingTargetPath:/csi/staging/hml_redis/rw-file-system-single-node-writer VolumeCapability:mount:<> access_mode:<mode:SINGLE_NODE_WRITER >  Secrets:map[] VolumeContext:map[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0426 21:56:54.554617       1 node.go:157] NodeStageVolume: volume="vol-0998a88f677e69072" operation finished
I0426 21:56:54.554631       1 inflight.go:69] Node Service: volume="vol-0998a88f677e69072" operation finished
E0426 21:56:54.554636       1 driver.go:119] GRPC error: rpc error: code = InvalidArgument desc = Device path not provided
I0426 21:56:54.562009       1 node.go:407] NodeUnpublishVolume: called with args {VolumeId:vol-0998a88f677e69072 TargetPath:/csi/per-alloc/48be08a6-4487-2ed6-7359-665e73c74993/hml_redis/rw-file-system-single-node-writer XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0426 21:56:54.562049       1 node.go:425] NodeUnpublishVolume: unmounting /csi/per-alloc/48be08a6-4487-2ed6-7359-665e73c74993/hml_redis/rw-file-system-single-node-writer
I0426 21:56:54.564069       1 node.go:421] NodeUnPublishVolume: volume="vol-0998a88f677e69072" operation finished
I0426 21:56:54.564082       1 inflight.go:69] Node Service: volume="vol-0998a88f677e69072" operation finished
E0426 21:56:54.564087       1 driver.go:119] GRPC error: rpc error: code = Internal desc = Could not unmount "/csi/per-alloc/48be08a6-4487-2ed6-7359-665e73c74993/hml_redis/rw-file-system-single-node-writer": unmount failed: exit status 32
Unmounting arguments: /csi/per-alloc/48be08a6-4487-2ed6-7359-665e73c74993/hml_redis/rw-file-system-single-node-writer
Output: umount: /csi/per-alloc/48be08a6-4487-2ed6-7359-665e73c74993/hml_redis/rw-file-system-single-node-writer: no mount point specified.
I0426 21:56:54.564370       1 node.go:236] NodeUnstageVolume: called with args {VolumeId:vol-0998a88f677e69072 StagingTargetPath:/csi/staging/hml_redis/rw-file-system-single-node-writer XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0426 21:56:54.564569       1 node.go:268] [Debug] NodeUnstageVolume: /csi/staging/hml_redis/rw-file-system-single-node-writer target not mounted
I0426 21:56:54.564575       1 node.go:251] NodeUnStageVolume: volume="vol-0998a88f677e69072" operation finished
I0426 21:56:54.564579       1 inflight.go:69] Node Service: volume="vol-0998a88f677e69072" operation finished
I0426 21:57:11.448828       1 node.go:498] NodeGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0426 21:57:41.449613       1 node.go:498] NodeGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0426 21:58:11.450222       1 node.go:498] NodeGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0426 21:58:41.450881       1 node.go:498] NodeGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0426 21:59:11.452806       1 node.go:498] NodeGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0426 21:59:41.453470       1 node.go:498] NodeGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0426 22:00:11.454233       1 node.go:498] NodeGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}