Monitoring and Alerting of Hashicorp Consul and Vault?

Team,

I have a cluster of 6 windows 2012 R2 VM’s per environment(test/stage/prod total is 18) in which a structure consul and vault is setup.

  1. 4 servers has consul hosted as server
  2. 2 servers each have consul(as client) + vault hosted

Now I want to enable monitoring and alerting mechanism on this structure. Few people told me to use a combination of grafana, prometheus etc. but I’m pretty new with all this.

Could you please suggest me the pattern like from where I can start reading or what are the tools, database etc. needed to establish a complete cluster monitoring command center.

Thank you

Hi @smartaquarius10
This is a difficult question to start to answer, because it seems like there is quite a distance between where you are starting and where you’d like to end up, i.e. a fully-configured monitoring system for your setup.

I think it would be beneficial to split the problem into two :

  1. How do I set up a monitoring, alerting and visualisation system (e.g. a prometheus data scraper + grafana data visualisation)
  2. How do I instrument the applications I want to monitor

The first part needs its own topic, I think, but let’s assume you get a Prometheus instance up and running. Now, it needs to scrape the things you want to monitor in order to get data to analyse and generate alerts from, as well as send to Grafana to visualise.

Luckily, both Consul and Vault are already instrumented for this kind of monitoring. The general idea is that the application exports metrics that can be used by the scraper.

I would start with the Consul tutorial:

Metrics are enabled by the prometheus_retention_time.

Similarly, Vault metric scraping can be enabled according to the prometheus telemetry configuration.

There is also a tutorial for Vault monitoring:

The latter might give you a good overview since it goes into some detail about how to set up the collection and visualisation components as well.

1 Like

Thanks a lot for detailed information. Just one more query do we need to setup any DB also or prometheus can use the disk space to save the logs. Because let say’ if I need to set any alert using grafana then data has to be saved somewhere so that interrogation will go on internally.

Currently, I have 2 red hat machines where I can setup prometheus and grafana but only one MS SQL database is available at the moment not any NOSQL one. But, for MS SQL db also I have to raise a purchase request so that is why just avoiding it.