Nomad ship logs to ELK

Nomad
1

Hi there.

I’m trying to define a system job running filebeat to ship the logs to a central location. I’ve searched around here and there were a custom filbeat image implementing the autodiscover for nomad. It as been merged and it’s now available in the official filebeat image. Autodiscover | Filebeat Reference [7.15] | Elastic

But I can’t get it to actually collect the logs. I’m using this job template, running as a system job :

filebeat.autodiscover:
  providers:
    - type: nomad
      node: {{ env "node.unique.name" }}
      address: http://{{ env "attr.unique.network.ip-address" }}:4646
      secret_id: "REDACTED"
      hints.enabled: true
      allow_stale: true
      templates:
        - condition:
            equals:
              nomad.namespace: "default"
          config:
            - type: log
              paths:
                - /var/nomad/alloc/${data.meta.alloc_id}/alloc/logs/${data.meta.task.name}.stderr.[0-9]*
                - /var/nomad/alloc/${data.meta.alloc_id}/alloc/logs/${data.meta.task.name}.stdout.[0-9]*
              ignore_older: 24h
output.logstash:
  hosts: ['https://REDACTED:8200']
  ssl:
    verification_mode: 'none'

Do someone have a clue on how to ship logs from nomad using filebeat ?

2

Hello Did you get any solution? I am stuck on same the problem

3

Hi, I found this while debugging something else. We’re successfully shipping logs from Nomad to ELK with Filebeat.

This is our config (/etc/filebeat/filebeat.yml) related to the Nomad autodiscover:

filebeat.autodiscover.providers:
  - type: nomad
    address: http://127.0.0.1:4646
    scope: node
    hints.enabled: true
    templates:
    - config:
      - type: log
        paths:
          - "/opt/nomad/data/alloc/${data.nomad.allocation.id}/alloc/logs/${data.meta.nomad.task.name}.stderr.[0-9]*"
          - "/opt/nomad/data/alloc/${data.nomad.allocation.id}/alloc/logs/${data.meta.nomad.task.name}.stdout.[0-9]*"

We’re running Filebeat 8.3.3 and Nomad 1.3.2.

If you’re not sure why your setup is not logging, make sure the paths are correct, and set the filebeat log level to debug.

4

Hi!

How did you solve the volume problem? In my nomad job i mount volume

volume_mount {
        volume      = "nomad_data"
        destination = "/mnt/data/nomad/alloc/"
        read_only   = false
      }

but can’t collect data by this path
/mnt/data/nomad/alloc/*/alloc/logs/*.stderr.[0-9]* , just because user filebeat don’t have enough permissions :

filebeat@d39d8da5b681:~$ ls /mnt/data/nomad/alloc/
ls: cannot open directory '/mnt/data/nomad/alloc/': Permission denied
5

Any updates about this case? Thanks!