I’m running a bunch of containers in nomad that are built with a log-shipper sidecar container following the pattern outlined here: https://nomadproject.io/guides/operating-a-job/accessing-logs/#log-shipper-pattern
When the primary container finishes its task (or if it fails), there’s a grace period for the sidecar to finish up log processing before also terminating. I’ve run into a few instances, however, where the sidecar has failed, which in turn causes my primary task/container to terminate.
Is there a way to allow the primary task/container to continue running if the sidecar fails? While I want my logs to be as complete as possible, losing a log shipper is far less impactful than having a batch job crash mid-way through processing its data set. I’ve seen articles that show how to do this in k8s, but I’m not sure how to do this in nomad. Help?