Logging collection in docker and let datadog agent to collect to display in datadog

Hi dear,

We run Hashicorp’s Nomad/Consul service mesh framework and collect metrics using Datadog. This works now

We also want to collect logs from Nomad itself and from jobs that run inside Nomad. For example, we have a Java program that Nomad runs in a Docker container and we would like those logs to be visible in Datadog as well.

Could you help give us more detail step by step? Very appreciate!

Hi @AccelaMichaelHe,

Unfortunately we do not currently have a Learn guide on the subject of log shipping, however, I will try and give an overview.

collect logs from Nomad itself

Assuming the Nomad agent is running as a host service, you could run the DataDog Agent host log shipper to collect logs from the host machine and send these to DataDog.

from jobs that run inside Nomad

There are a couple of different ways in which to achieve this. This first would be to run the DataDog agent in the same manner as described above with enough permissions to read the contents of the allocation directory. The allocation directory holds a number of items, including the stdout and stderr output of the running application. The Nomad filesystem concepts page has more information on this exact layout.

Another way would be to have a log shipping sidecar application per container you are running. This would involve running the DataDog agent as a container task within the same task group that your Java program is running, tailing the files described above. The advantage here is that the sidecar is isolated to the allocation directory structure and cannot ready anything else from the host. This does come at a resource utilisation cost though, as each container you wish to ship logs from would require its own log shipping sidecar.

The final way would be to utilise the Docker driver logging task configuration options to configure a Docker logging driver. This final method is specific to Docker and will stop certain Nomad functionality, such as viewing allocations logs via the CLI or UI.

Thanks,
jrasell and the Nomad team

Thanks Jrasell. Let me try and may get back to you if I have any question.
Very appreciate!

Dear Jrasell,

We plan to use the third solution you provided, before that we want to double confirm is the DD agent hosted or container based?

Based on your comment posted, we have set up the docker default logging driver (json-file) and also we can see the STDOUT logs in the docker (docker logs {container id}), and the next thing we still got confused, is this enough or we still need more configuration on it?

I saw there is a custom logging file include a config.yml, for this solution, do we need this?

Very appreciate!

Regards,
Michael

Hi @AccelaMichaelHe,

the DD agent hosted or container based

I believe it can run either as a host process or within a container as a sidecar process; how you wish to run it depends on your requirements.

is this enough or we still need more configuration on it

This again depends on your requirements; typically to use a sidecar or host based log shipper you should not need to any more Docker configuration that is set as default by the Nomad task driver.

a custom logging file include a config.yml

Could you link to what this is in reference to please?

Thanks,
jrasell and the Nomad team

Dear @jrasell ,

Per the third solution you provided
The final way would be to utilise the Docker driver [logging task configuration options](https://developer.hashicorp.com/nomad/docs/drivers/docker#logging-1) to configure a [Docker logging driver](https://docs.docker.com/config/containers/logging/configure/). This final method is specific to Docker and will stop certain Nomad functionality, such as viewing allocations logs via the CLI or UI.

We are thinking DD Agent is kind of an instance job on nomad so that can grab different kind of application services logs to Datadog.

Service A
Service B. ->. DD Agent container base in nomad collect A, B , C to display to DD
Service C

From your latest comment, it seems it mostly similar like the solution 2 which need run the DD agent within the same task group embed with an application service like below ?
Service A - DD agent embed and collect Service A logs
Service B - DD agent embed and collect Service B logs

Here is some reference for a custom logging file include a config.yml

Looking forward to hearing from you.
Regards,
Michael

Dear @jrasell ,

May I know do you have any idea or suggestion above?

Regards,
Michael