Nomad Recent events logging

Hi Team,

We wanted to capture the nomad recent events logs that are shown in Nomad UI to our logging. Currently we don’t find any logs in nomad logs stored in system.

This is to keep a track of which service gets restarted and due to what error.

Could you please guide how we can get that setup.

We are running nomad v0.12.7

Thanks

Hi @surajthakur,

The Nomad clients are responsible for logging these types of activities. I have performed a local check and the logs you’re interested in are logged at both INFO and DEBUG, therefore if your log level is INFO or above you may not see all the lines as shown below. You could therefore setup a log shipper to process the Nomad client logs and ship these off to your central log store for capture.

    2020-11-30T09:10:02.301+0100 [DEBUG] client: allocation updates applied: added=1 removed=0 updated=0 ignored=0 errors=0
    2020-11-30T09:10:02.303+0100 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: starting plugin: alloc_id=f00f95b9-6cc1-c582-ed18-3dffc25faddf task=redis path=/Users/jrasell/go/bin/nomad args=[/Users/jrasell/go/bin/nomad, logmon]
    2020-11-30T09:10:02.305+0100 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: plugin started: alloc_id=f00f95b9-6cc1-c582-ed18-3dffc25faddf task=redis path=/Users/jrasell/go/bin/nomad pid=83724
    2020-11-30T09:10:02.305+0100 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: waiting for RPC address: alloc_id=f00f95b9-6cc1-c582-ed18-3dffc25faddf task=redis path=/Users/jrasell/go/bin/nomad
    2020-11-30T09:10:02.314+0100 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon.nomad: plugin address: alloc_id=f00f95b9-6cc1-c582-ed18-3dffc25faddf task=redis @module=logmon address=/var/folders/sx/83shrk2s7bs09xvn51vpsxkr0000gn/T/plugin352590995 network=unix timestamp=2020-11-30T09:10:02.314+0100
    2020-11-30T09:10:02.314+0100 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: using plugin: alloc_id=f00f95b9-6cc1-c582-ed18-3dffc25faddf task=redis version=2
    2020-11-30T09:10:02.316+0100 [INFO]  client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=f00f95b9-6cc1-c582-ed18-3dffc25faddf task=redis @module=logmon path=/private/var/folders/sx/83shrk2s7bs09xvn51vpsxkr0000gn/T/NomadClient525622947/f00f95b9-6cc1-c582-ed18-3dffc25faddf/alloc/logs/.redis.stdout.fifo timestamp=2020-11-30T09:10:02.316+0100
    2020-11-30T09:10:02.316+0100 [INFO]  client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=f00f95b9-6cc1-c582-ed18-3dffc25faddf task=redis @module=logmon path=/private/var/folders/sx/83shrk2s7bs09xvn51vpsxkr0000gn/T/NomadClient525622947/f00f95b9-6cc1-c582-ed18-3dffc25faddf/alloc/logs/.redis.stderr.fifo timestamp=2020-11-30T09:10:02.316+0100
    2020-11-30T09:10:02.343+0100 [DEBUG] client.driver_mgr.docker: image reference count incremented: driver=docker image_name=redis:3.2 image_id=sha256:87856cc39862cec77541d68382e4867d7ccb29a85a17221446c857ddaebca916 references=1
    2020-11-30T09:10:02.344+0100 [DEBUG] client.driver_mgr.docker: configured resources: driver=docker task_name=redis memory=268435456 memory_reservation=0 cpu_shares=500 cpu_quota=0 cpu_period=0
    2020-11-30T09:10:02.344+0100 [DEBUG] client.driver_mgr.docker: binding directories: driver=docker task_name=redis binds="[]string{"/private/var/folders/sx/83shrk2s7bs09xvn51vpsxkr0000gn/T/NomadClient525622947/f00f95b9-6cc1-c582-ed18-3dffc25faddf/alloc:/alloc", "/private/var/folders/sx/83shrk2s7bs09xvn51vpsxkr0000gn/T/NomadClient525622947/f00f95b9-6cc1-c582-ed18-3dffc25faddf/redis/local:/local", "/private/var/folders/sx/83shrk2s7bs09xvn51vpsxkr0000gn/T/NomadClient525622947/f00f95b9-6cc1-c582-ed18-3dffc25faddf/redis/secrets:/secrets"}"
    2020-11-30T09:10:02.344+0100 [DEBUG] client.driver_mgr.docker: networking mode not specified; using default: driver=docker task_name=redis
    2020-11-30T09:10:02.344+0100 [DEBUG] client.driver_mgr.docker: allocated static port: driver=docker task_name=redis ip=127.0.0.1 port=30197
    2020-11-30T09:10:02.344+0100 [DEBUG] client.driver_mgr.docker: exposed port: driver=docker task_name=redis port=30197
    2020-11-30T09:10:02.344+0100 [DEBUG] client.driver_mgr.docker: applied labels on the container: driver=docker task_name=redis labels=map[com.hashicorp.nomad.alloc_id:f00f95b9-6cc1-c582-ed18-3dffc25faddf]
    2020-11-30T09:10:02.344+0100 [DEBUG] client.driver_mgr.docker: setting container name: driver=docker task_name=redis container_name=redis-f00f95b9-6cc1-c582-ed18-3dffc25faddf
    2020-11-30T09:10:02.501+0100 [DEBUG] client: updated allocations: index=13 total=1 pulled=0 filtered=1
    2020-11-30T09:10:02.501+0100 [DEBUG] client: allocation updates: added=0 removed=0 updated=0 ignored=1
    2020-11-30T09:10:02.501+0100 [DEBUG] client: allocation updates applied: added=0 removed=0 updated=0 ignored=1 errors=0
    2020-11-30T09:10:02.533+0100 [INFO]  client.driver_mgr.docker: created container: driver=docker container_id=df3277df8cf1009120efac91e06b1171ccde049fe55634629da37fd130668564
    2020-11-30T09:10:03.255+0100 [INFO]  client.driver_mgr.docker: started container: driver=docker container_id=df3277df8cf1009120efac91e06b1171ccde049fe55634629da37fd130668564

In the upcoming 1.0.0 release, the Nomad events stream feature may be of interest to you which allows users subscribe to a single unified timeline that streams all high-level events. A good overview of this can be found on the HashiCorp Blog.

jrasell and the Nomad team.

1 Like

Thanks @jrasell. I will try that. We are also testing out the 1.0.0 release.