Hetzner Cloud (hcloud) CSI missing access and attachment mode

Nomad
1

Nomad: 1.3.1

Deployed this CSI job on hetzner cloud (hcloud) results in a healthy CSI plugin:

variable "datacenters" {
  type    = list(string)
  default = ["dc1"]
}
variable "namespace" {
  type    = string
  default = "default"
}

job "hcloud-csi" {
  datacenters = var.datacenters
  namespace   = var.namespace
  type        = "system"

  group "node" {
    task "plugin" {
      driver = "docker"

      config {
        image      = "hetznercloud/hcloud-csi-driver:1.6.0"
        privileged = true
      }

      env {
        CSI_ENDPOINT = "unix:///csi/csi.sock"
        HCLOUD_TOKEN = "..."
      }

      csi_plugin {
        id        = "csi.hetzner.cloud"
        type      = "monolith"
        mount_dir = "/csi"
      }
    }
  }
}

then I successfully created a volume (verified in the hcloud UI)

id           = "vol1"
namespace    = "default"
name         = "vol1"
type         = "csi"
plugin_id    = "csi.hetzner.cloud"
capacity_min = "10Gi"
capacity_max = "10Gi"

capability {
  access_mode     = "single-node-writer"
  attachment_mode = "file-system"
}

mount_options {
  fs_type     = "ext4"
  mount_flags = ["noatime"]
}

 $ hcloud volume list
ID         NAME   SIZE    SERVER   LOCATION
20204336   vol1   10 GB   -        fsn1

 $ hcloud volume describe -o json 20204336 | jq
{
  "created": "2022-06-07T22:10:06+00:00",
  "format": null,
  "id": 20204336,
  "labels": {},
  "linux_device": "/dev/disk/by-id/scsi-0HC_Volume_20204336",
  "location": {
    "city": "Falkenstein",
    "country": "DE",
    "description": "Falkenstein DC Park 1",
    "id": 1,
    "latitude": 50.47612,
    "longitude": 12.370071,
    "name": "fsn1",
    "network_zone": "eu-central"
  },
  "name": "vol1",
  "protection": {
    "delete": false
  },
  "server": null,
  "size": 10,
  "status": "available"
}

this results in:

 $ nomad volume status                 
Container Storage Interface
ID    Name  Plugin ID          Schedulable  Access Mode
vol1  vol1  csi.hetzner.cloud  true         <none>

there I already see the missing access mode. When I tried to use the volume in a job:

job "httpd1" {
  datacenters = ["dc1"]

  group "httpd" {
    restart {
      attempts = 10
      interval = "5m"
      delay    = "25s"
      mode     = "delay"
    }
    volume "vol1" {
      type      = "csi"
      read_only = false
      source    = "vol1"
    }

    task "httpd" {
      driver = "docker"

      volume_mount {
        volume      = "vol1"
        destination = "/srv"
        read_only   = false
      }

      config {
        image = "httpd:2.4.41-alpine"
      }
    }
  }
}

I get an error

 $ nomad job run test/job.nomad                  
Error submitting job: Unexpected response code: 500 (1 error occurred:
        * Task group httpd validation failed: 1 error occurred:
        * Task group volume validation for vol1 failed: 2 errors occurred:
        * CSI volumes must have an attachment mode
        * CSI volumes must have an access mode





)

any hints, what is going wrong?

2

Hey there!

I’m fiddling with CSI plugin on Hetzner as well, definitely not an expert here, but I’m not seeing external_id in your volume declaration (HCL file)

type = "csi"
id = "vol1"
name = "vol1"
external_id = "<id of the volume from hetzner cloud>"
# ... the rest of the file

any chance you missed that? (not sure if it’s even possible to register volume without that attribute, just guessing; edit: guess based on the fact that I have exact same setup with same Nomad version - and aside some long-running quirks - it seems to work)

3

thanks @rwojsznis,

if I run nomad volume create vol1.hcl the volume gets created and I can see its external ID from hcloud in nomad

 $ nomad volume status vol1            
ID                   = vol1
Name                 = vol1
External ID          = 20220267
Plugin ID            = csi.hetzner.cloud
Provider             = csi.hetzner.cloud
Version              = 1.6.0
Schedulable          = true
Controllers Healthy  = 3
Controllers Expected = 3
Nodes Healthy        = 3
Nodes Expected       = 3
Access Mode          = <none>
Attachment Mode      = <none>
Mount Options        = fs_type: ext4 flags: [REDACTED]
Namespace            = default

Topologies
Topology  Segments
01        csi.hetzner.cloud/location=fsn1

Allocations
No allocations placed

So I assume external ID is only needed if you have an existing volume on hcloud and using register to assign it to nomad?

4

@rwojsznis do you have set access mode like this? or is this different then yours?

capability {
  access_mode     = "single-node-writer"
  attachment_mode = "file-system"
}
5

if I run nomad volume create vol1.hcl the volume gets created and I can see its external ID from hcloud in nomad

Hmm, it shows 20220267. Yet in the first post you mentioned:

hcloud volume list
ID         NAME   SIZE    SERVER   LOCATION
20204336   vol1   10 GB   -        fsn1

Can you register volume and explicitly specify external_id to match hetzner’s volume id? (20204336)

I will post my config later today but it is almost identical to yours :see_no_evil:

6

the IDs don’t much anymore because I keep trying to debug it and keep deleting/recreating the volume :wink: but they do match.

 $ hcloud volume list                     
ID         NAME   SIZE    SERVER   LOCATION
20220267   vol1   10 GB   -        fsn1

 $ nomad volume status vol1             
ID                   = vol1
Name                 = vol1
External ID          = 20220267

I also run volume register with the external_id which works but does not change anything

7

the IDs don’t much anymore because I keep trying to debug it and keep deleting/recreating the volume

Oh, right, roger that!

In that case I’m kinda clueless here :confused: here is my volume config:

type = "csi"
id = "volume-mixed"
name = "volume-mixed"
external_id = "<id from hetzner>"
plugin_id = "csi.hetzner.cloud"

capability {
  access_mode     = "single-node-writer"
  attachment_mode = "file-system"
}

mount_options {
  fs_type     = "ext4"
  mount_flags = ["discard" ,"defaults"]
}
8

Could you share what nomad volume status volume-mixed says about access_mode and attachment_mode?

9

Sure thing!

root@nomad-1:~# nomad volume status volume-mixed
ID                   = volume-mixed
Name                 = volume-mixed
External ID          = <id from hetzner>
Plugin ID            = csi.hetzner.cloud
Provider             = csi.hetzner.cloud
Version              = 1.6.0
Schedulable          = true
Controllers Healthy  = 3
Controllers Expected = 3
Nodes Healthy        = 3
Nodes Expected       = 3
Access Mode          = single-node-writer
Attachment Mode      = file-system
Mount Options        = fs_type: ext4 flags: [REDACTED]
Namespace            = default

Allocations
ID        Node ID   Task Group         Version  Desired  Status   Created    Modified
c4786841  33bc1695  authelia-services  0        run      running  4h19m ago  4h15m ago

I hope that somewhat helps :sweat_smile:

edit: did you tried checking logs from CSI plugin or Nomad itself? :thinking:

10

@resmo im running into the same issue. Did you ever figure it out?

11

@sarendsen1 good to know it’s is not related to my setup. No, haven’t had time yet to look at it.

12

@resmo figured it out. Need to set the capability on the network block in the job aswell.

See capability.mdx

and you must provide a block for each capability you intend to use in a job’s volume block

example:

    volume "vol" {
      type      = "csi"
      read_only = false
      source    = "vol1"
      attachment_mode = "file-system"
      access_mode     = "single-node-writer"
      per_alloc       = false
    }
3 Likes
13

Thanks @sarendsen1, I was able to get it running with your advise!

14

Note the 2.0.0 of the plugin introduced breaking changes and your job needs to be adjusted: Tested 2.0.0-rc.3 on Nomad, test failed · Issue #330 · hetznercloud/csi-driver · GitHub