[SERF] How to get the metrics from Serf on Prometheus and Grafana?

Consul
1

Dear All,

Hope you all are doing fine.

I have a question on the possibility of visualizing the serf metrics in Prometheus and Grafana. I have 100+ nodes which has serf binary. What I need to achieve is getting serf metrics - basically to understand what is happening when a member is joined or removed, time it takes a single join node to stabilize, any other crucial metrics - and show them visually using Prometheus and Grafana.

Also, I read in a paper called “network coordinates in the wild” that nodes have a tendency to keep drifting away to a direction from the source (in vivaldi), so I also need to see how this works in serf.

I also found serf/docs/agent/telemetry.html.markdown at master · hashicorp/serf · GitHub Additionally, I came across GitHub - hashicorp/go-metrics: A Golang library for exporting performance and runtime metrics to external metrics systems (i.e. statsite, statsd).

However, I do not understand how to integrate either or how does it work :cry: What I simply need is to expose the serf metrics to Grafana.

I am a working with serf for the first time and totally new to this type of work. Therefore, I would sincerely grateful for any guidance or resources that I can refer to make things clear.

Thank you!

1 Like
2

I was able to sort this out with statsd.

Anyway, I am posting here what needs to be done just to save some time and effort for anyone who faces the same issue.

Serf will stream various telemetry information to either statsd or statsite. This can be used to capture various runtime information.

In serf client configuration we need to set statsd_addr parameter with the address of running statsd instance. If you need to capture metrics from serf from a container make sure the container is able to access the IP and Port of statsd instance.

For Prometheus, it is needed to use a statsd exporter. In statsd exporter, we need to map the metrics so it knows how to convert StatsD metrics into Prometheus metrics. Once you configure statsd, Prometheus, then grafana can be connected.

I hope this helps!