How to add custom metadata to nodes without restarting nomad?

Nomad
1

Hello,

It is possible to add custom labels to a node before running jobs so they can be used as constraints? Nomad documentation explains you can:

Now here is the problem: I see than the metadata is defined on the client section of the nomad configuration file (not the jobs):

client {
  enabled       = true
  network_speed = 500
  node_class    = "prod"

  meta {
    "owner" = "ops"
    "cached_binaries" = "redis,apache,nginx,jq,cypress,nodejs"
    "rack" = "rack-12-1"
  }
}

So this means I need to hardcode this information then restart my nodes?
There is a way to update this information dynamically on the client without a reboot?

For the sake of argument, this is how you do it in Kubernetes using kubectl:

kubectl get nodes --namespace=default --show-labels
for node in company1.company.com company2.company.com; do
        case $node in
                company1.company.com)
                        kubectl label nodes $node product="ProLiant-XXX-Gen666" --overwrite
                        kubectl label nodes $node fast_switch="yes" --overwrite
                        ;;
                company2.company.com)
                        kubectl label nodes $node product="ProLiant-ZZZZ-Gen777" --overwrite
                        kubectl label nodes $node fast_switch="no" --overwrite
                        ;;
                *)
                        ;;
        esac
done
kubectl get nodes --namespace=default --show-labels

Please let me know if I misunderstood the way this works.

Thanks!

2

I’ve been having this same issue and haven’t found a solution

3

I also have the same issue. Do you have a solution?

4

Hi all :wave:

Sorry for missing this message.

This is currently not possible as the client metadata is sent to servers only when the agent starts.

Would you mind opening a feature request so we can better track this?

Thank you!

3 Likes
5

When restarting nomad service after changing e.g. the metadata in the client config (manually, ansible, salt, etc), the jobs on the node will not be restarted, if I remember correctly.

Not sure about your use-case here. You just want to update metadata without stopping nodes/jobs, or you want to add metadata when certain jobs are started on a node?

6

Hi folks.

Would this be the relevant issue in the nomad GitHub project to watch for updates about annotating nodes with custom data?

With the new release of 1.4, I wondered if it might be plausible to use the new key value features in nomad secure variables as a stopgap until there was API support for annotating nodes directly.

7

Hi @mrchrisadams,

Would this be the relevant issue in the nomad GitHub project to watch for updates about annotating nodes with custom data?

Yes that issue is the correct one to use for updates from engineers when this work is picked up.

use the new key value features in nomad secure variables as a stopgap until there was API support for annotating nodes directly

This seems like an interesting idea and could be possible, however, I would urge caution particularly around the rate of change of entries. Each entry is committed via Raft and so incurs CPU, network, and disk overheads for all servers in your cluster. If you do look into this and have any feedback, please let us know.

Thanks,
jrasell and the Nomad team

1 Like