As some metrics are only available on cluster leader node, I wanted to ask you what is the best way to know who is the leader. We are using prometheus grafana for monitoring.
There is a metric called consul_raft_state_leader which changes when the node become cluster leader. But I didn’t see how to use it in prometheus to ensure that the node is the leader. There is also http endpoint ‘/v1/status/leader’ which provides exactly what I need - current cluster leader. But I’m not sure how to user it right in grafana. Could you please share how you do you monitor cluster leader specific metrics, for example ‘consul_autopilot_healthy’ metric.
Did you try this expression in prometheus?
Yes, I did. This is cumulative metric, and increments each time the node became cluster leader. In our case it looks like this currently
I thought how we can use it to know exactly who is the leader in each point in time. I suppose we should keep somewhere previous value to compare it with current value and in case it changed we should change the leader in our metrics requests accordingly. But I cant understand is there a way to do this im PromQL?
You could have a look at this: