Nomad CSI plugin / volume register fail

Nomad
1

I have been following the Nomad CSI learn tutorial but I am not able to complete this, I am getting the following error when registering the volume.hcl

root@ip-10-10-16-192:/var/snap/amazon-ssm-agent/2012# nomad volume register volume.hcl
Error registering volume: Unexpected response code: 500 (validate volume: controller plugin returned an internal error, check the plugin allocation logs for m
ore information: rpc error: code = Internal desc = Could not get volume with ID "vol-0861b29c6e638db36": RequestError: send request failed
caused by: Post "https://ec2.us-west-2.amazonaws.com/": dial tcp: lookup ec2.us-west-2.amazonaws.com on 10.10.16.192:53: read udp 172.17.0.2:37751->10.10.16.1
92:53: i/o timeout)

this is my plugin output

root@ip-10-10-16-192:/var/snap/amazon-ssm-agent/2012# nomad plugin status aws-ebs0
ID                   = aws-ebs0
Provider             = ebs.csi.aws.com
Version              = v0.6.0
Controllers Healthy  = 1
Controllers Expected = 1
Nodes Healthy        = 5
Nodes Expected       = 5

Allocations
ID        Node ID   Task Group  Version  Desired  Status   Created    Modified
cbbd7157  5dbc4dfc  controller  0        run      running  6h51m ago  6h50m agob10fc142  94090d8b  nodes       0        run      running  32m5s ago  31m37s ago
9c198f72  aa3b0673  nodes       0        run      running  6h48m ago  6h48m ago
3c17566f  5dbc4dfc  nodes       0        run      running  6h48m ago  6h48m ago1c30d4ee  a9de2dd1  nodes       0        run      running  6h48m ago  6h48m ago
286a5de8  2695141c  nodes       0        run      running  32m8s ago  31m49s ago

this are the controller container logs

I0915 22:39:00.635747       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0915 22:39:29.752356       1 controller.go:336] ValidateVolumeCapabilities: called with args {VolumeId:vol-0861b29c6e638db36 VolumeContext:map[] VolumeCapabilities:[mount:<> access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[] Secrets:map[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0915 22:39:30.637296       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
E0915 22:39:50.103875       1 driver.go:111] GRPC error: rpc error: code = Internal desc = Could not get volume with ID "vol-0861b29c6e638db36": RequestError: send request failed
caused by: Post "https://ec2.us-west-2.amazonaws.com/": dial tcp: lookup ec2.us-west-2.amazonaws.com on 10.10.16.192:53: read udp 172.17.0.2:41352->10.10.16.192:53: i/o timeout
I0915 22:40:00.639304       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0915 22:40:30.640461       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0915 22:41:00.641788       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0915 22:41:30.642732       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0915 22:41:57.985500       1 controller.go:336] ValidateVolumeCapabilities: called with args {VolumeId:vol-0861b29c6e638db36 VolumeContext:map[] VolumeCapabilities:[mount:<> access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[] Secrets:map[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0915 22:42:00.645182       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
E0915 22:42:18.333102       1 driver.go:111] GRPC error: rpc error: code = Internal desc = Could not get volume with ID "vol-0861b29c6e638db36": RequestError: send request failed
caused by: Post "https://ec2.us-west-2.amazonaws.com/": dial tcp: lookup ec2.us-west-2.amazonaws.com on 10.10.16.192:53: read udp 172.17.0.2:39181->10.10.16.192:53: read: connection refused
I0915 22:42:30.646892       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0915 22:43:00.648640       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0915 22:43:30.650010       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0915 22:44:00.651935       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0915 22:44:30.652815       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0915 22:45:00.653634       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0915 22:45:30.654807       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0915 22:46:00.656331       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0915 22:46:30.658264       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0915 22:47:00.659374       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0915 22:47:30.660846       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0915 22:47:42.227088       1 controller.go:336] ValidateVolumeCapabilities: called with args {VolumeId:vol-0861b29c6e638db36 VolumeContext:map[] VolumeCapabilities:[mount:<> access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[] Secrets:map[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0915 22:48:00.663073       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
E0915 22:48:02.587542       1 driver.go:111] GRPC error: rpc error: code = Internal desc = Could not get volume with ID "vol-0861b29c6e638db36": RequestError: send request failed
caused by: Post "https://ec2.us-west-2.amazonaws.com/": dial tcp: lookup ec2.us-west-2.amazonaws.com on 10.10.16.192:53: read udp 172.17.0.2:37751->10.10.16.192:53: i/o timeout
I0915 22:48:30.664441       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}

am I missing some configuration ?? EC2 instances are able to communicate to internet, and ping the ec2 endpoint

getting volume with aws cli

root@ip-10-10-16-192:/var/snap/amazon-ssm-agent/2012# AWS_REGION="us-west-2" aws ec2 describe-volumes --volume-ids vol-0861b29c6e638db36
{
    "Volumes": [
        {
            "Attachments": [],
            "AvailabilityZone": "us-west-2a",
            "CreateTime": "2020-09-14T23:35:23.612Z",
            "Encrypted": false,
            "Size": 40,
            "SnapshotId": "",
            "State": "available",
            "VolumeId": "vol-0861b29c6e638db36",
            "Iops": 120,
            "VolumeType": "gp2",
            "MultiAttachEnabled": false
        }
    ]
}

this will discard iam issues

2

Hi @Crizstian! This error message is the important one:

The plugin is looking up the AWS API endpoint DNS entry and failing to make the connection to the DNS server. So there’s something in the DNS configuration of either the plugin job, Docker, or the host that’s not letting the plugin make that DNS query.