Hi dear,
We run Hashicorp’s Nomad/Consul service mesh framework and collect metrics using Datadog. This works now
We also want to collect logs from Nomad itself and from jobs that run inside Nomad. For example, we have a Java program that Nomad runs in a Docker container and we would like those logs to be visible in Datadog as well.
Could you help give us more detail step by step? Very appreciate!
Hi @AccelaMichaelHe,
Unfortunately we do not currently have a Learn guide on the subject of log shipping, however, I will try and give an overview.
collect logs from Nomad itself
Assuming the Nomad agent is running as a host service, you could run the DataDog Agent host log shipper to collect logs from the host machine and send these to DataDog.
from jobs that run inside Nomad
There are a couple of different ways in which to achieve this. This first would be to run the DataDog agent in the same manner as described above with enough permissions to read the contents of the allocation directory. The allocation directory holds a number of items, including the stdout and stderr output of the running application. The Nomad filesystem concepts page has more information on this exact layout.
Another way would be to have a log shipping sidecar application per container you are running. This would involve running the DataDog agent as a container task within the same task group that your Java program is running, tailing the files described above. The advantage here is that the sidecar is isolated to the allocation directory structure and cannot ready anything else from the host. This does come at a resource utilisation cost though, as each container you wish to ship logs from would require its own log shipping sidecar.
The final way would be to utilise the Docker driver logging task configuration options to configure a Docker logging driver. This final method is specific to Docker and will stop certain Nomad functionality, such as viewing allocations logs via the CLI or UI.
Thanks,
jrasell and the Nomad team
Thanks Jrasell. Let me try and may get back to you if I have any question.
Very appreciate!
Dear Jrasell,
We plan to use the third solution you provided, before that we want to double confirm is the DD agent hosted or container based?
Based on your comment posted, we have set up the docker default logging driver (json-file) and also we can see the STDOUT logs in the docker (docker logs {container id}), and the next thing we still got confused, is this enough or we still need more configuration on it?
I saw there is a custom logging file include a config.yml, for this solution, do we need this?
Very appreciate!
Regards,
Michael
Hi @AccelaMichaelHe,
the DD agent hosted or container based
I believe it can run either as a host process or within a container as a sidecar process; how you wish to run it depends on your requirements.
is this enough or we still need more configuration on it
This again depends on your requirements; typically to use a sidecar or host based log shipper you should not need to any more Docker configuration that is set as default by the Nomad task driver.
a custom logging file include a config.yml
Could you link to what this is in reference to please?
Thanks,
jrasell and the Nomad team
Dear @jrasell ,
Per the third solution you provided
The final way would be to utilise the Docker driver [logging task configuration options](https://developer.hashicorp.com/nomad/docs/drivers/docker#logging-1) to configure a [Docker logging driver](https://docs.docker.com/config/containers/logging/configure/). This final method is specific to Docker and will stop certain Nomad functionality, such as viewing allocations logs via the CLI or UI.
We are thinking DD Agent is kind of an instance job on nomad so that can grab different kind of application services logs to Datadog.
Service A
Service B. ->. DD Agent container base in nomad collect A, B , C to display to DD
Service C
From your latest comment, it seems it mostly similar like the solution 2 which need run the DD agent within the same task group embed with an application service like below ?
Service A - DD agent embed and collect Service A logs
Service B - DD agent embed and collect Service B logs
Here is some reference for a custom logging file include a config.yml
Looking forward to hearing from you.
Regards,
Michael
@jrasell May I know do you have any idea on it?
Hi @AccelaMichaelHe,
From your latest comment, it seems it mostly similar like the solution 2 which need run the DD agent within the same task group embed with an application service like below ?
Yes this looks like the sidecar based approach where each allocation would have a dedicated DD agent to ship logs and provides a secure approach at the cost of being slightly resource heavy at it requires a DD agent per allocation.
Another approach would be to run the DD agent as a Nomad system job which results in a single agent per host. This agent would need access to the host filesystem in order to access the allocation log files within the Nomad data directory. It therefore comes at the cost of some security (depending on your requirements) at the benefit of less resource usage overall and a potentially easier to manage setup.
Thanks,
jrasell and the Nomad team
Just chiming in, I previously ran this last approach you mentioned so I have an example for how it was done at least 
I will reiterate the security implication on this one though due to accessing data within the nomad data dir, however, in my case I’ve mostly been experimenting in a home lab environment so I was ok with this security implication but i may choose differently with another setup
The repo is using promtail and loki for log shipping and management, however the concept would be very similar to the datadog one afaik and is more for showing how you could get the logs into the agent/shipper CarbonCollins - Cloud / Monitoring / Log Agent · GitLab
I did also run sidecar log shippers at one point too and quite liked that solution as it kept the service isolation and did not need to have the nomad data dir mounted, i was however running on some very underpowered hardware so ran onto resource issues 
if you have the resources though it’s imho the better solution over the host nomad dir mounting solution.
Thanks @CarbonCollins @jrasell !
Hi @jrasell ,
Hope you are doing well.
Our team also want to try the second approach:
Another way would be to have a log shipping sidecar application per container you are running. This would involve running the DataDog agent as a container task within the same task group that your Java program is running, tailing the files described above. The advantage here is that the sidecar is isolated to the allocation directory structure and cannot ready anything else from the host. This does come at a resource utilisation cost though, as each container you wish to ship logs from would require its own log shipping sidecar.
But still confused with what is the sidecar application mean? Could you give us an example?
Best Regards,
Michael