I am using consul 1.8.3 (non enterprise) version for my testing environment to monitor service health on my 2 node cluster (Windows & Linux).
The consul cluster setup is done as server-client, Linux node plays the server role. I have various data systems (DB2, Memsql, ES, Solr, Hbase etc) running on my Linux node. For each of these data systems I have service checks, both TCP/Port ping check and custom script checks.
Recently I started getting issues. Whenever a service is unhealthy, I see the consul debug monitor doesn’t show the threshold failure/success count (I have defined both success_before_passing & failures_before_critical). Though, it prints out the individual check as critical.
This is causing an issue as I have a dependent recovery script, which should initiate when the threshold crosses over the limit defined in configs (json).
Is there any way I can figure out, why consul service is not counting the threshold or behind the scene, if anything I have to check to make sure once the threshold limit is reached, it picks up the script which is defined to get triggered on FAILURE/CRITICAL state?
Also, where can I see the consul logs?
Thanks & Regards,
–RB