Help with ceph CSI woes

As promised during the last nomad community hours I have rebuilt my hashistack cluster with cph volume support so everything should be clear and default. I am running into similar issues where a task get rescheduled but keeps # allocs so it will show as volume exhausted even though there are no running allocs. I thought I could get over this by creating and mounting my volumes with multi-node-single writer and that seemed to have worked mostly except not I have a job that rescheduled but still has a ‘completed alloc’ that uses up the write allocation.

Im not sure what additional information to give but I figure mounting as a block device for multi node multi write is prob not the right way to handle this. And I also know I can manually detach and these allocs go away and it starts working but there has to be a better way.

PS I already forgot who to tag from the stream but effort is somewhere!