So, I’ve a logging setup for
docker driver (thanks to this post) and get logs with enriched metadata.
However, my questions are more for
raw_exec driver. I don’t seem to find enough docs/config options to get this up and running myself, hence asking here.
Currently, Nomad stores the stdout/stderr logs inside
alloc/logs/ directory. If the task is executing as
exec, there’s no way to access these logs from the host. The typical solution of running a log collection agent (like
vector) as a system job and read logs from a common directory (like
/var/logs) isn’t possible, because Nomad currently doesn’t provide the option to configure a logging directory.
So, is the only choice to run a log collection agent as a
sidecar task, which has access to
/alloc/logs directory and collect it from there?
EDIT: I can also run
vector as a
system job and collect logs from
/opt/nomad/data/allocs/ folder. But the biggest issue here is that I won’t know which particular
group the alloc belongs to, unless I query Nomad API somehow before collecting the logs.
This seems a bit too hard than it should be Just wondering if I am missing out on something
They way I approached this problem - also loosely based on the blog post you mention - I just run system promtail job that scrapes everything I’m interested in and pushes it to Grafana cloud.
Because I use consul I grab the services that have
promtail=true (chosen arbitrarily), grab the ID of the task from the consul (
_nomad-task-* part) and slurp
alloc/logs/ directory (I mount system’s
/var/nomad/alloc directory on the job in
It’s probably not the most elegant solution but works for me
This week I hope to do the small write-up on the subject - if you’re interested in implementation details I can reach out to you (don’t want to spam the forum with external resources of my own making - will check the forum rules in that matter )
I’ve been using nomad-vector-logger in production since sometime. It’s working well for my usecase that I described above.
The daemon runs alongside
vector (same group, different tasks so that they can share the same
alloc_dir). It periodically fetches a list of allocations on the current node with their filepath and templates out a
vector config. The user can provide their own extra
vector templates to configure extra transformation (JSON/logfmt parsing/splitting namespaces etc) and then finally route to sink providers (Clickhouse/Elastic/S3 etc).
When the allocs are stopped, this daemon also removes them from the templated file after a configurable delay period, to ensure that Vector has finished processing all the remaining logs.
If anyone is facing an issue with exec/raw_exec tasks logging, check this out Feedback welcome!