Csi aws ebs plugin topology failure

Nomad
1

hello,

i’m trying to use a ebs volume created with the following file :

id        = "miID"
name      = "357cb1a-2a11-400e-8f88-545d9d172039" # CSIVolumeName tag, must be idempotent
type      = "csi"
plugin_id = "aws-ebs0"

capacity_min = "10GiB"
capacity_max = "50GiB"

capability {
  access_mode     = "single-node-writer"
  attachment_mode = "file-system"
}

capability {
  access_mode     = "single-node-writer"
  attachment_mode = "block-device"
}

parameters {
  type = "gp2"
}

topology_request {
  required {
    topology {
      segments {
        "topology.ebs.csi.aws.com/zone" = "eu-west-3a"
      }
    }
  }
}

the volume is created successfully, but when i try to use it in a job i get the following error

"did not meet topology requirement": 3 nodes excluded by filter

by checking my plugin configuration it looks like this topology should be available :

ID                   = aws-ebs0
Provider             = ebs.csi.aws.com
Version              = v1.29.1
Controllers Healthy  = 1
Controllers Expected = 1
Nodes Healthy        = 3
Nodes Expected       = 3

Controller Capabilities
  ATTACH_READONLY
  CLONE_VOLUME
  CONTROLLER_ATTACH_DETACH
  CREATE_DELETE_SNAPSHOT
  CREATE_DELETE_VOLUME
  EXPAND_VOLUME
  GET_CAPACITY
  GET_VOLUME
  LIST_SNAPSHOTS
  LIST_VOLUMES
  LIST_VOLUMES_PUBLISHED_NODES
  VOLUME_CONDITION

Node Capabilities
  EXPAND_VOLUME
  GET_VOLUME_STATS
  STAGE_UNSTAGE_VOLUME
  VOLUME_ACCESSIBILITY_CONSTRAINTS
  VOLUME_CONDITION

Accessible Topologies
Node ID   Accessible Topology
35352a1c  topology.kubernetes.io/zone=eu-west-3a,kubernetes.io/os=linux,topology.ebs.csi.aws.com/zone=eu-west-3a
71e0ecbd  kubernetes.io/os=linux,topology.ebs.csi.aws.com/zone=eu-west-3b,topology.kubernetes.io/zone=eu-west-3b
cb681d27  topology.ebs.csi.aws.com/zone=eu-west-3c,topology.kubernetes.io/zone=eu-west-3c,kubernetes.io/os=linux

Allocations
ID                                    Eval ID                               Node ID                               Node Name          Task Group  Version  Desired  Status   Created                    Modified
50f72dff-6cf8-79a1-303f-845a380300e6  0ec286c4-fef6-66fa-6f32-3ddc6ae27eff  cb681d27-9e27-56ef-1934-5455c8c1f0b0  ip-192-168-62-116  controller  3        run      running  2024-04-25T22:15:22+02:00  2024-04-25T22:15:33+02:00
debf1de5-13cf-d18c-444a-174f573d185b  d3293e55-1db6-8af2-49be-eb444fbd0117  cb681d27-9e27-56ef-1934-5455c8c1f0b0  ip-192-168-62-116  nodes       7        run      running  2024-04-25T22:14:58+02:00  2024-04-25T22:15:24+02:00
30a8a900-8eae-c5fc-b7fe-7e94323d8199  d3293e55-1db6-8af2-49be-eb444fbd0117  71e0ecbd-aef8-3ed6-623a-8fe78eff2712  ip-192-168-52-115  nodes       7        run      running  2024-04-25T22:14:58+02:00  2024-04-25T22:15:24+02:00
ebadec9f-c843-115c-78f1-423b18aeefa8  d3293e55-1db6-8af2-49be-eb444fbd0117  35352a1c-6636-ad1c-5ade-ce07359932ea  ip-192-168-42-114  nodes       7        run      running  2024-04-25T22:14:58+02:00  2024-04-25T22:15:23+02:00

Can you please help me to troubleshoot this ?

2

you are hitting this bug: Fix CSITopology matching logic for csi storage driver by thefallentree · Pull Request #24522 · hashicorp/nomad · GitHub

the workaround is to use v1.26.1 of aws-ebs-csi driver until this patch is merged.

1 Like