Is there any way for us disable or change the frequency of the nomad snapshots?
We have a couple of thousand task groups each with one task inside them that is running a batch job. The batch job can take a while to run which seems to result in excessive memory growth on the nomad servers.
We went from scheduling 180,000 jobs each with 1 taskgroup and 1 task in them (ran out of nomad server memory after a few hours, UI unusable), to 5 jobs with 500 taskgroups each with 72 batch task in them (ran out of nomad server memory after a few hours), to 5 jobs with 500 taskgroups each with 1 batch task in them, running a bash script containing 72 sequential command line commands (ran out of nomad server memory after a few hours and disk space due to the 48gb raft db. I suspect the memory may have been in part due to it trying to take a snapshot while running out of disk space).
The later approach got us further, but still memory grows on the nomad servers to over 100gb. I am now starting to think it is growing when it is trying to take a snapshot and would like to disable the snapshots to see if it helps.
If we cant disable them, is it safe to continually delete them?
Cheers
Jonathan