Ebs csi driver - csi_hook failed: Device path not provided

Nomad
1

On running mysql test job for the CSI plugin seeing an error on the csi hook, job is unable to start. Any ideas what causes this error?

failed to setup alloc: pre-run hook “csi_hook” failed: rpc error: code = InvalidArgument desc = Device path not provided

Nomad version 1.0.4

nomad volume status
Container Storage Interface
ID     Name   Plugin ID  Schedulable  Access Mode
mysql  mysql  aws-ebs0   true         single-node-write

nomad plugin status
Container Storage Interface
ID        Provider         Controllers Healthy/Expected  Nodes Healthy/Expected
aws-ebs0  ebs.csi.aws.com  1/1                           6/6

nomad plugin status aws-ebs0
ID                   = aws-ebs0
Provider             = ebs.csi.aws.com
Version              = v0.10.0
Controllers Healthy  = 1
Controllers Expected = 1
Nodes Healthy        = 6
Nodes Expected       = 6

Allocations
ID        Node ID   Task Group  Version  Desired  Status   Created     Modified
8f3a4ef8  d0db8e71  nodes       0        run      running  17h44m ago  17h44m ago
24194094  e61b5ec4  controller  0        run      running  17h44m ago  17h44m ago
e568be37  e61b5ec4  nodes       0        run      running  17h44m ago  17h44m ago
c5029653  e6a4987d  nodes       0        run      running  17h44m ago  17h44m ago
593fa020  182b8e7f  nodes       0        run      running  17h44m ago  17h44m ago
3808a9d3  9c0c772d  nodes       0        run      running  17h44m ago  17h44m ago
20f1535a  a24a0b58  nodes       0        run      running  17h44m ago  17h44m ago

using an admin token to launch the job

job "mysql-server" {
  datacenters = ["${nomad_datacenter}"]
  type        = "service"

  group "mysql-server" {
    count = 1

    volume "mysql" {
      type      = "csi"
      read_only = false
      source    = "mysql"
    }

    network {
      port "db" {
        static = 3306
      }
    }

    restart {
      attempts = 10
      interval = "5m"
      delay    = "25s"
      mode     = "delay"
    }

    task "mysql-server" {
      driver = "docker"

      volume_mount {
        volume      = "mysql"
        destination = "/srv"
        read_only   = false
      }

      env {
        MYSQL_ROOT_PASSWORD = "password"
      }

      config {
        image = "hashicorp/mysql-portworx-demo:latest"
        args  = ["--datadir", "/srv/mysql"]
        ports = ["db"]
      }

      resources {
        cpu    = 500
        memory = 1024
      }

      service {
        name = "mysql-server"
        port = "db"

        check {
          type     = "tcp"
          interval = "10s"
          timeout  = "2s"
        }
      }
    }
  }
}
2

It looks like this might be where the error is coming from

4

logs from the node that got the request

I0406 20:39:37.197872       1 node.go:104] NodeStageVolume: called with args {VolumeId:vol-06beef729a2725546 PublishContext:map[] StagingTargetPath:/csi/staging/mysql/rw-file-system-single-node-writer VolumeCapability:mount:<fs_type:"ext4" > access_mode:<mode:SINGLE_NODE_WRITER >  Secrets:map[] VolumeContext:map[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0406 20:39:37.197944       1 node.go:152] NodeStageVolume: volume="vol-06beef729a2725546" operation finished
I0406 20:39:37.197956       1 inflight.go:69] Node Service: volume="vol-06beef729a2725546" operation finished
E0406 20:39:37.197961       1 driver.go:118] GRPC error: rpc error: code = InvalidArgument desc = Device path not provided
I0406 20:39:37.202322       1 node.go:381] NodeUnpublishVolume: called with args {VolumeId:vol-06beef729a2725546 TargetPath:/csi/per-alloc/7c9d05e0-2f96-9c62-fb5b-88bd418c01d3/mysql/rw-file-system-single-node-writer XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0406 20:39:37.202370       1 node.go:399] NodeUnpublishVolume: unmounting /csi/per-alloc/7c9d05e0-2f96-9c62-fb5b-88bd418c01d3/mysql/rw-file-system-single-node-writer
I0406 20:39:37.205425       1 node.go:395] NodeUnPublishVolume: volume="vol-06beef729a2725546" operation finished
I0406 20:39:37.205443       1 inflight.go:69] Node Service: volume="vol-06beef729a2725546" operation finished
E0406 20:39:37.205450       1 driver.go:118] GRPC error: rpc error: code = Internal desc = Could not unmount "/csi/per-alloc/7c9d05e0-2f96-9c62-fb5b-88bd418c01d3/mysql/rw-file-system-single-node-writer": unmount failed: exit status 32
Unmounting arguments: /csi/per-alloc/7c9d05e0-2f96-9c62-fb5b-88bd418c01d3/mysql/rw-file-system-single-node-writer
Output: umount: /csi/per-alloc/7c9d05e0-2f96-9c62-fb5b-88bd418c01d3/mysql/rw-file-system-single-node-writer: no mount point specified.
I0406 20:39:37.205870       1 node.go:210] NodeUnstageVolume: called with args {VolumeId:vol-06beef729a2725546 StagingTargetPath:/csi/staging/mysql/rw-file-system-single-node-writer XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0406 20:39:37.206123       1 node.go:242] NodeUnstageVolume: /csi/staging/mysql/rw-file-system-single-node-writer target not mounted
I0406 20:39:37.206137       1 node.go:225] NodeUnStageVolume: volume="vol-06beef729a2725546" operation finished
I0406 20:39:37.206145       1 inflight.go:69] Node Service: volume="vol-06beef729a2725546" operation finished
5

This Nomad issue seems related (although fixed in Nomad 0.11.2):

Maybe @tgross can help?

6

Another thing I noticed is that the EBS plugin is requiring that PublishContext[DevicePathKey] exists, while nomad does not do any validation on PublishContext that I can see.

7

Have you resolved this? We have the same issue basically we cannot attach an EBS drive to a job. Same errors.

version nomad 1.0.4
csi plugin 0.10.1

8

@jessequinn yes, the issue for us was extra additional AWS permission were needed for the EBS driver during creation stage. If you are managing permissions with tag filters, make sure those tags are getting added on creation. Otherwise, you might next to dive into cloud trail. The error is surely misleading.