Hello, I have a nomad cluster on AWS whose job has a CSI volume and I set a CloudWatch alarm to check disk full. In the case of the root device, specifying the resource to
/ is enough.
However, in the case of CSI volumes, I do not know even what is the best way to see the used percentage of the volume.
nomad volume status -verbose <job_name> print nothing about that.
nomad alloc exec -job <job_name> df gives the information, but I don’t know how to tell the metric to ClouwWatch agent.
$ nomad alloc exec -job mysql sh -c "df -h"
Filesystem Size Used Avail Use% Mounted on
overlay 16G 9.9G 6.2G 62% /
tmpfs 64M 0 64M 0% /dev
tmpfs 977M 0 977M 0% /sys/fs/cgroup
/dev/nvme0n1p1 16G 9.9G 6.2G 62% /alloc
tmpfs 1.0M 0 1.0M 0% /secrets
/dev/nvme1n1 7.9G 5.3G 2.6G 68% /srv/mysql
shm 64M 0 64M 0% /dev/shm
tmpfs 977M 0 977M 0% /proc/acpi
tmpfs 977M 0 977M 0% /proc/scsi
tmpfs 977M 0 977M 0% /sys/firmware
1: Manually Create or Edit the CloudWatch Agent Configuration File - Amazon CloudWatch
What CSI driver are you using? I believe there may be monitoring methods depending which implementation you are running.
I do not know the exact answer currently so will do some research when possible. It does seem though that the CloudWatch agent exports disk metrics including utilisation.
jrasell and the Nomad team
I solved the problem. my configuration JSON is a little wired but works, it looks like this:
The resource pattern is found by the below command:
$ df -a | grep mysql
/dev/nvme1n1 - - - - /opt/nomad/client/csi/node/aws-ebs0/staging/mysql/rw-file-system-single-node-writer
/dev/nvme1n1 - - - - /opt/nomad/client/csi/node/aws-ebs0/per-alloc/681acfc8-ff0f-a9c7-89f5-3842454176e3/mysql/rw-file-system-single-node-writer
/opt/nomad/client/csi/node/aws-ebs0/per-alloc/681acfc8-ff0f-a9c7-89f5-3842454176e3/mysql/rw-file-system-single-node-writer is also valid, but its arbitrary path is not desired to use.