AWS EBS CSI node job failing

Nomad
1

I am trying to learn how to implement CSI plugins and have persistent storage in nomad.
I have a single nomad-node client setup, where I am running two jobs one for plugin controller & plugin node.

volume.hcl

type      = "csi"
id        = "<id>"
plugin_id = "aws-ebs0"
external_id = "<aws volume id>"
 
name = "test"
 
capability {
  access_mode     = "single-node-writer"
  attachment_mode = "file-system"
}

controller.hcl

job "plugin-aws-ebs-controller" {
  datacenters = ["dc1"]
 
  group "controller" {
    task "plugin" {
      driver = "docker"
 
      config {
        image = "public.ecr.aws/ebs-csi-driver/aws-ebs-csi-driver:v1.25.0"
 
        args = [
          "controller",
          "--endpoint=unix://csi/csi.sock",
          "--logtostderr",
          "--v=5"
        ]
      }
      env {
        AWS_ACCESS_KEY_ID     = "<my aws access key>"
        AWS_SECRET_ACCESS_KEY = "<my aws secret key>"
        AWS_REGION            = "eu-west-3"
      }
 
      csi_plugin {
        id        = "aws-ebs0"
        type      = "controller"
        mount_dir = "/csi"
      }
 
      resources {
        cpu    = 500
        memory = 256
      }
    }
  }
}

node.hcl

job "plugin-aws-ebs-nodes" {
  datacenters = ["dc1"]
 
  type = "system"
 
  group "nodes" {
    task "plugin" {
      driver = "docker"
 
      config {
        image = "public.ecr.aws/ebs-csi-driver/aws-ebs-csi-driver:v1.25.0"
 
        args = [
          "node",
          "--endpoint=unix://csi/csi.sock",
          "--logtostderr",
          "--v=5",
        ]
 
        privileged = true
      }
      env {
        AWS_ACCESS_KEY_ID     = ""
        AWS_SECRET_ACCESS_KEY = ""
        AWS_REGION            = "eu-west-3"
      }
 
      csi_plugin {
        id        = "aws-ebs0"
        type      = "node"
        mount_dir = "/csi"
      }
 
      resources {
        cpu    = 500
        memory = 256
      }
    }
  }
}

I am running this jobs on the client, where the job for controller is running fine, volume & plugin are registered.

Screenshot 2023-12-03 at 2.50.32 PM
Screenshot 2023-12-03 at 2.50.32 PM1286×112 7.57 KB

the volume is attached. (only it states that Schedulable = false)

I am getting this error logs in node task:

I1203 19:36:51.986890       1 driver.go:78] "Driver Information" Driver="ebs.csi.aws.com" Version="v1.25.0"
I1203 19:36:51.987644       1 node.go:82] "[Debug] Retrieving node info from metadata service"
I1203 19:36:51.987805       1 node.go:84] "regionFromSession Node service" region="eu-west-3"
I1203 19:36:51.988099       1 metadata.go:85] "retrieving instance data from ec2 metadata"
I1203 19:37:04.572882       1 metadata.go:88] "ec2 metadata is not available"
I1203 19:37:04.572937       1 metadata.go:96] "retrieving instance data from kubernetes api"
I1203 19:37:04.573009       1 metadata.go:99] "error creating kubernetes api client" err="unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined"
panic: error getting instance data from ec2 metadata or kubernetes api

goroutine 1 [running]:
github.com/kubernetes-sigs/aws-ebs-csi-driver/pkg/driver.newNodeService(0xc0001aefc0)
	/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/pkg/driver/node.go:87 +0x3cc
github.com/kubernetes-sigs/aws-ebs-csi-driver/pkg/driver.NewDriver({0xc0006c7ed0, 0xb, 0x4?})
	/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/pkg/driver/driver.go:100 +0x3e6
main.main()
	/go/src/github.com/kubernetes-sigs/aws-ebs-csi-driver/cmd/main.go:64 +0x532

The task deployment logs say:

Error: CSI plugin failed probe: timeout while connecting to gRPC socket: failed to stat socket: stat /etc/nomad.d/data/client/csi/plugins/91a89dcf-eb89-a956-193e-074d6933f612/csi.sock: no such file or directory

I am using my custom VPS which is hosted on prem.
Operating System: Ubuntu
Nomad Version: v1.6.3

2

Hi @shreejilucifer,

The aws-ebs-csi-driver CSI driver expects to be running in AWS or on a Kubernetes cluster, so it can perform this setup functionality. In the case of running on Nomad, it expects to be able to find the EC2 Metadata service which it cannot as you’re running within a custom on-prem VPS.

Thanks,
jrasell and the Nomad team

1 Like