Nomad CSI plugin / volume register fail

I have been following the Nomad CSI learn tutorial but I am not able to complete this, I am getting the following error when registering the volume.hcl

root@ip-10-10-16-192:/var/snap/amazon-ssm-agent/2012# nomad volume register volume.hcl
Error registering volume: Unexpected response code: 500 (validate volume: controller plugin returned an internal error, check the plugin allocation logs for m
ore information: rpc error: code = Internal desc = Could not get volume with ID "vol-0861b29c6e638db36": RequestError: send request failed
caused by: Post "https://ec2.us-west-2.amazonaws.com/": dial tcp: lookup ec2.us-west-2.amazonaws.com on 10.10.16.192:53: read udp 172.17.0.2:37751->10.10.16.1
92:53: i/o timeout)

this is my plugin output

root@ip-10-10-16-192:/var/snap/amazon-ssm-agent/2012# nomad plugin status aws-ebs0
ID                   = aws-ebs0
Provider             = ebs.csi.aws.com
Version              = v0.6.0
Controllers Healthy  = 1
Controllers Expected = 1
Nodes Healthy        = 5
Nodes Expected       = 5

Allocations
ID        Node ID   Task Group  Version  Desired  Status   Created    Modified
cbbd7157  5dbc4dfc  controller  0        run      running  6h51m ago  6h50m agob10fc142  94090d8b  nodes       0        run      running  32m5s ago  31m37s ago
9c198f72  aa3b0673  nodes       0        run      running  6h48m ago  6h48m ago
3c17566f  5dbc4dfc  nodes       0        run      running  6h48m ago  6h48m ago1c30d4ee  a9de2dd1  nodes       0        run      running  6h48m ago  6h48m ago
286a5de8  2695141c  nodes       0        run      running  32m8s ago  31m49s ago

this are the controller container logs

I0915 22:39:00.635747       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0915 22:39:29.752356       1 controller.go:336] ValidateVolumeCapabilities: called with args {VolumeId:vol-0861b29c6e638db36 VolumeContext:map[] VolumeCapabilities:[mount:<> access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[] Secrets:map[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0915 22:39:30.637296       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
E0915 22:39:50.103875       1 driver.go:111] GRPC error: rpc error: code = Internal desc = Could not get volume with ID "vol-0861b29c6e638db36": RequestError: send request failed
caused by: Post "https://ec2.us-west-2.amazonaws.com/": dial tcp: lookup ec2.us-west-2.amazonaws.com on 10.10.16.192:53: read udp 172.17.0.2:41352->10.10.16.192:53: i/o timeout
I0915 22:40:00.639304       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0915 22:40:30.640461       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0915 22:41:00.641788       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0915 22:41:30.642732       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0915 22:41:57.985500       1 controller.go:336] ValidateVolumeCapabilities: called with args {VolumeId:vol-0861b29c6e638db36 VolumeContext:map[] VolumeCapabilities:[mount:<> access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[] Secrets:map[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0915 22:42:00.645182       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
E0915 22:42:18.333102       1 driver.go:111] GRPC error: rpc error: code = Internal desc = Could not get volume with ID "vol-0861b29c6e638db36": RequestError: send request failed
caused by: Post "https://ec2.us-west-2.amazonaws.com/": dial tcp: lookup ec2.us-west-2.amazonaws.com on 10.10.16.192:53: read udp 172.17.0.2:39181->10.10.16.192:53: read: connection refused
I0915 22:42:30.646892       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0915 22:43:00.648640       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0915 22:43:30.650010       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0915 22:44:00.651935       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0915 22:44:30.652815       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0915 22:45:00.653634       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0915 22:45:30.654807       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0915 22:46:00.656331       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0915 22:46:30.658264       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0915 22:47:00.659374       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0915 22:47:30.660846       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0915 22:47:42.227088       1 controller.go:336] ValidateVolumeCapabilities: called with args {VolumeId:vol-0861b29c6e638db36 VolumeContext:map[] VolumeCapabilities:[mount:<> access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[] Secrets:map[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0915 22:48:00.663073       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
E0915 22:48:02.587542       1 driver.go:111] GRPC error: rpc error: code = Internal desc = Could not get volume with ID "vol-0861b29c6e638db36": RequestError: send request failed
caused by: Post "https://ec2.us-west-2.amazonaws.com/": dial tcp: lookup ec2.us-west-2.amazonaws.com on 10.10.16.192:53: read udp 172.17.0.2:37751->10.10.16.192:53: i/o timeout
I0915 22:48:30.664441       1 controller.go:310] ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}

am I missing some configuration ?? EC2 instances are able to communicate to internet, and ping the ec2 endpoint

getting volume with aws cli

root@ip-10-10-16-192:/var/snap/amazon-ssm-agent/2012# AWS_REGION="us-west-2" aws ec2 describe-volumes --volume-ids vol-0861b29c6e638db36
{
    "Volumes": [
        {
            "Attachments": [],
            "AvailabilityZone": "us-west-2a",
            "CreateTime": "2020-09-14T23:35:23.612Z",
            "Encrypted": false,
            "Size": 40,
            "SnapshotId": "",
            "State": "available",
            "VolumeId": "vol-0861b29c6e638db36",
            "Iops": 120,
            "VolumeType": "gp2",
            "MultiAttachEnabled": false
        }
    ]
}

this will discard iam issues