As a brief background on our setup, This question is regarding the session_ttl_min setting in consul. We have a product suite that comprises of consul, vault and nomad as the control plane services. Currently, we have not altered the session_ttl_min setting and is at 10s default. We also have postgres HA setup using Patroni HA tool which uses a key/value that’s created on consul and it’s associate session TTL to identify unhealthy nodes and trigger failover. Due to the current limit set on session_ttl_min, postgres failover time takes around 20s and more.
My Question: I do understand that this limit is configurable and can be reduced in order to solve my problem. But we are a little reluctant to alter this setting due to the recommendation by Hashicorp on the config doc "It is recommended to keep this limit at or above the default to encourage clients to send infrequent heartbeats". Just from this line I am not able to get a grasp of how this would affect us. I searched for documents around this and found none. Could you please help me understand how reducing session_ttl_min would affect consul or any other services using it? If there’s a counter setting that I can change to tackle such outcome, could you please point me to that.
Thanks
Pon Venkatesh M
VMware