Nomad ship logs to ELK

Hi there.

I’m trying to define a system job running filebeat to ship the logs to a central location. I’ve searched around here and there were a custom filbeat image implementing the autodiscover for nomad. It as been merged and it’s now available in the official filebeat image. Autodiscover | Filebeat Reference [7.15] | Elastic

But I can’t get it to actually collect the logs. I’m using this job template, running as a system job :

filebeat.autodiscover:
  providers:
    - type: nomad
      node: {{ env "node.unique.name" }}
      address: http://{{ env "attr.unique.network.ip-address" }}:4646
      secret_id: "REDACTED"
      hints.enabled: true
      allow_stale: true
      templates:
        - condition:
            equals:
              nomad.namespace: "default"
          config:
            - type: log
              paths:
                - /var/nomad/alloc/${data.meta.alloc_id}/alloc/logs/${data.meta.task.name}.stderr.[0-9]*
                - /var/nomad/alloc/${data.meta.alloc_id}/alloc/logs/${data.meta.task.name}.stdout.[0-9]*
              ignore_older: 24h
output.logstash:
  hosts: ['https://REDACTED:8200']
  ssl:
    verification_mode: 'none'

Do someone have a clue on how to ship logs from nomad using filebeat ?

Hello Did you get any solution? I am stuck on same the problem

Hi, I found this while debugging something else. We’re successfully shipping logs from Nomad to ELK with Filebeat.

This is our config (/etc/filebeat/filebeat.yml) related to the Nomad autodiscover:

filebeat.autodiscover.providers:
  - type: nomad
    address: http://127.0.0.1:4646
    scope: node
    hints.enabled: true
    templates:
    - config:
      - type: log
        paths:
          - "/opt/nomad/data/alloc/${data.nomad.allocation.id}/alloc/logs/${data.meta.nomad.task.name}.stderr.[0-9]*"
          - "/opt/nomad/data/alloc/${data.nomad.allocation.id}/alloc/logs/${data.meta.nomad.task.name}.stdout.[0-9]*"

We’re running Filebeat 8.3.3 and Nomad 1.3.2.

If you’re not sure why your setup is not logging, make sure the paths are correct, and set the filebeat log level to debug.

Hi!

How did you solve the volume problem? In my nomad job i mount volume

volume_mount {
        volume      = "nomad_data"
        destination = "/mnt/data/nomad/alloc/"
        read_only   = false
      }

but can’t collect data by this path
/mnt/data/nomad/alloc/*/alloc/logs/*.stderr.[0-9]* , just because user filebeat don’t have enough permissions :

filebeat@d39d8da5b681:~$ ls /mnt/data/nomad/alloc/
ls: cannot open directory '/mnt/data/nomad/alloc/': Permission denied