How to set CloudWatch Agent for metrics of disk attached by CSI

Nomad
1

Hello, I have a nomad cluster on AWS whose job has a CSI volume and I set a CloudWatch alarm to check disk full. In the case of the root device, specifying the resource to / is enough.[1]

However, in the case of CSI volumes, I do not know even what is the best way to see the used percentage of the volume. df or nomad volume status -verbose <job_name> print nothing about that.
Only nomad alloc exec -job <job_name> df gives the information, but I don’t know how to tell the metric to ClouwWatch agent.

$ nomad alloc exec -job mysql sh -c "df -h"
Filesystem      Size  Used Avail Use% Mounted on
overlay          16G  9.9G  6.2G  62% /
tmpfs            64M     0   64M   0% /dev
tmpfs           977M     0  977M   0% /sys/fs/cgroup
/dev/nvme0n1p1   16G  9.9G  6.2G  62% /alloc
tmpfs           1.0M     0  1.0M   0% /secrets
/dev/nvme1n1    7.9G  5.3G  2.6G  68% /srv/mysql
shm              64M     0   64M   0% /dev/shm
tmpfs           977M     0  977M   0% /proc/acpi
tmpfs           977M     0  977M   0% /proc/scsi
tmpfs           977M     0  977M   0% /sys/firmware

Thanks.

1: Manually Create or Edit the CloudWatch Agent Configuration File - Amazon CloudWatch

2

Hi @lens0021,

What CSI driver are you using? I believe there may be monitoring methods depending which implementation you are running.

I do not know the exact answer currently so will do some research when possible. It does seem though that the CloudWatch agent exports disk metrics including utilisation.

Thanks,
jrasell and the Nomad team

3

I solved the problem. my configuration JSON is a little wired but works, it looks like this:

{
  "metrics": {
    "metrics_collected": {
      "disk": {
        "measurement": [
          "used_percent"
        ],
        "resources": [
          "/",
          "/opt/nomad/client/csi/node/aws-ebs0/staging/<VOLUME_NAME>/rw-file-system-single-node-writer"
        ]
      }
    }
  }
}

The resource pattern is found by the below command:

$ df -a | grep mysql
/dev/nvme1n1             -        -         -    - /opt/nomad/client/csi/node/aws-ebs0/staging/mysql/rw-file-system-single-node-writer
/dev/nvme1n1             -        -         -    - /opt/nomad/client/csi/node/aws-ebs0/per-alloc/681acfc8-ff0f-a9c7-89f5-3842454176e3/mysql/rw-file-system-single-node-writer

/opt/nomad/client/csi/node/aws-ebs0/per-alloc/681acfc8-ff0f-a9c7-89f5-3842454176e3/mysql/rw-file-system-single-node-writer is also valid, but its arbitrary path is not desired to use.