I have jobs that write calculation results to files inside their
alloc directories. After they finish, other jobs can possibly read these result files (using
client HTTP API). The problem I noticed is that when
gc_max_allocs is exceeded, the
alloc directories can be deleted immediately after a job finishes, so the other jobs have no chance to read the results.
I didn’t find any option to specify the minimum time the
alloc directory is preserved. How can I do that? One option would be to keep the job running for some time after finishing, but it looks wasteful.
As you mention there is not a way to specify that Nomad keeps allocation directories around for a minimum time. All the garbage collection thresholds and intervals are designed to prevent unbounded growth; whereas a flurry of jobs and allocations with minimum preserved times could cause the servers to become excessively loaded.
Would it be possible to write the calculation results to an external source for consumption, such as AWS S3or GCP Cloud Storage? If not, how about attaching a volume to the calculation task which can then be mounted onto subsequent tasks?
jrasell and the Nomad team
Thank you for answering,
yes, we could use external storage but it involves extra management (like expiration) and adds delays. I think that for now I’ll make the jobs to sleep for some time after producing a result file.