What does Nomad System GC actually do? Is it ever necessary?

Nomad
1

Hey all,

Trying to determine if there are any benefits to ever manually running nomad system gc endpoint/command calls.

What does it actually do? Is there any difference between running it on a server or a client? I know there’s timed garbage collection, but over time does anything get to the point where a forced gc has potential benefits?

Thank you!

1 Like
2

Hi @chuckyz,

During normal Nomad operation its unlikely you’ll need to manually run this, instead relying on the periodic Nomad GC runner. Answering the questions directly:

  • What does it actually do? - It removes items from the internal Nomad Raft state which no longer need to be tracked. These objects include jobs, allocations, evaluations, deployments, CSI plugins, CSI volumes and nodes.
  • Is there any difference between running it on a server or a client? - The HTTP call will always be forwarded to the cluster leader as it only affects that state held by Nomad servers.
  • … forced gc has potential benefits? In path to production environments which have a high cadence of change, more regular GC can help keep the state size smaller also meaning CLI outputs and API returns are smaller. It also means the GC run itself will take a shorter period of time on each run. In these situations, Nomad provides configuration options (nodeGC, deploymentGC…) for controlling how and when the GC is run, so you don’t have to manually trigger the API/CLI.

I hope this helps. Please let me know if you have any followup questions.

Thanks,
jrasell and the Nomad team.

3

@chuckyz

We call a forced GC pretty regularly. We have a high cadence of windows node changes where the entire node is replaced. If we don’t do GC we end up with a bunch of harmless, yet cluttering, “left” nodes in queries so we actually use the GC to clean that up.

I’m curious - are you experiencing any issues calling GC, or some other scenario?

Thanks
Ian