Vault HA cluster with consul backend, frequent leader flip

Vault leader getting flipped quite frequently due to below ‘failed to acquire lock’ error from back end consul.

There are no abnormal logs from qconsul nodes.

2020-08-27T07:30:37.724Z [DEBUG] replication.index.perf: saved checkpoint: num_dirty=0
2020-08-27T07:30:38.520Z [WARN] core: leadership lost, stopping active operation
2020-08-27T07:30:38.520Z [INFO] core: pre-seal teardown starting
2020-08-27T07:30:39.023Z [WARN] core.cluster-listener: no TLS config found for ALPN: ALPN=[req_fw_sb-act_v1]
2020-08-27T07:30:39.023Z [DEBUG] core.cluster-listener: error handshaking cluster connection: error=“unsupported protocol”
2020-08-27T07:30:39.520Z [INFO] core: stopping replication
2020-08-27T07:30:39.520Z [INFO] core: closed sync connection
2020-08-27T07:30:39.520Z [TRACE] replication.perf.logshipper: interrupting streams
2020-08-27T07:30:39.520Z [TRACE] replication.perf.logshipper: done interrupting streams
2020-08-27T07:30:39.520Z [TRACE] replication.dr.logshipper: interrupting streams
2020-08-27T07:30:39.520Z [TRACE] replication.dr.logshipper: done interrupting streams
2020-08-27T07:30:39.520Z [INFO] core: replication stopped
2020-08-27T07:30:39.520Z [DEBUG] audit: removing reload function: path=file/
2020-08-27T07:30:39.520Z [DEBUG] expiration: stop triggered
2020-08-27T07:30:39.742Z [DEBUG] expiration: finished stopping
2020-08-27T07:30:39.742Z [INFO] rollback: stopping rollback manager
2020-08-27T07:30:39.755Z [DEBUG] replication.index.perf: saved checkpoint: num_dirty=0
2020-08-27T07:30:39.765Z [DEBUG] replication.index.local: saved checkpoint: num_dirty=0
2020-08-27T07:30:39.765Z [DEBUG] sealwrap: stopping upgrades
2020-08-27T07:30:39.770Z [DEBUG] core: set storage to read-write
2020-08-27T07:30:39.770Z [INFO] core: pre-seal teardown complete
2020-08-27T07:30:39.801Z [ERROR] core: failed to acquire lock: error=“failed to create session: Unexpected response code: 500 (rpc error: rpc error making call: Check ‘serfHealth’ is in critical state)”