Vault leader getting flipped quite frequently due to below ‘failed to acquire lock’ error from back end consul.
There are no abnormal logs from qconsul nodes.
2020-08-27T07:30:37.724Z [DEBUG] replication.index.perf: saved checkpoint: num_dirty=0
2020-08-27T07:30:38.520Z [WARN] core: leadership lost, stopping active operation
2020-08-27T07:30:38.520Z [INFO] core: pre-seal teardown starting
2020-08-27T07:30:39.023Z [WARN] core.cluster-listener: no TLS config found for ALPN: ALPN=[req_fw_sb-act_v1]
2020-08-27T07:30:39.023Z [DEBUG] core.cluster-listener: error handshaking cluster connection: error=“unsupported protocol”
2020-08-27T07:30:39.520Z [INFO] core: stopping replication
2020-08-27T07:30:39.520Z [INFO] core: closed sync connection
2020-08-27T07:30:39.520Z [TRACE] replication.perf.logshipper: interrupting streams
2020-08-27T07:30:39.520Z [TRACE] replication.perf.logshipper: done interrupting streams
2020-08-27T07:30:39.520Z [TRACE] replication.dr.logshipper: interrupting streams
2020-08-27T07:30:39.520Z [TRACE] replication.dr.logshipper: done interrupting streams
2020-08-27T07:30:39.520Z [INFO] core: replication stopped
2020-08-27T07:30:39.520Z [DEBUG] audit: removing reload function: path=file/
2020-08-27T07:30:39.520Z [DEBUG] expiration: stop triggered
2020-08-27T07:30:39.742Z [DEBUG] expiration: finished stopping
2020-08-27T07:30:39.742Z [INFO] rollback: stopping rollback manager
2020-08-27T07:30:39.755Z [DEBUG] replication.index.perf: saved checkpoint: num_dirty=0
2020-08-27T07:30:39.765Z [DEBUG] replication.index.local: saved checkpoint: num_dirty=0
2020-08-27T07:30:39.765Z [DEBUG] sealwrap: stopping upgrades
2020-08-27T07:30:39.770Z [DEBUG] core: set storage to read-write
2020-08-27T07:30:39.770Z [INFO] core: pre-seal teardown complete
2020-08-27T07:30:39.801Z [ERROR] core: failed to acquire lock: error=“failed to create session: Unexpected response code: 500 (rpc error: rpc error making call: Check ‘serfHealth’ is in critical state)”