Ensure alloc directories stay for some time

Hi.

I have jobs that write calculation results to files inside their alloc directories. After they finish, other jobs can possibly read these result files (using client HTTP API). The problem I noticed is that when gc_max_allocs is exceeded, the alloc directories can be deleted immediately after a job finishes, so the other jobs have no chance to read the results.

I didn’t find any option to specify the minimum time the alloc directory is preserved. How can I do that? One option would be to keep the job running for some time after finishing, but it looks wasteful.

Hi @aartur;

As you mention there is not a way to specify that Nomad keeps allocation directories around for a minimum time. All the garbage collection thresholds and intervals are designed to prevent unbounded growth; whereas a flurry of jobs and allocations with minimum preserved times could cause the servers to become excessively loaded.

Would it be possible to write the calculation results to an external source for consumption, such as AWS S3or GCP Cloud Storage? If not, how about attaching a volume to the calculation task which can then be mounted onto subsequent tasks?

Thanks,
jrasell and the Nomad team

Thank you for answering,

yes, we could use external storage but it involves extra management (like expiration) and adds delays. I think that for now I’ll make the jobs to sleep for some time after producing a result file.