Consul servers and agents went to hung state

Consul
1

we have consul cluster of 3 servers. As part of testing, we have done below steps.
1.our team has stopped consul1 and consul2 same time @13:48, with this consul3 became leader.
2. started consul2 @13:57.
3. stopped consul3@13:59, now the leader is consul2.
4.at 14:22 we have started consul1 and consul3. Here consul1 became the leader before the replication from consul2 ->consul1.
5.stopped consul1 @14:53 and restarted it @15:40

after we stopped consul@14:53, we have observed that no consul functionalities are responding, including writing log, health checks and any events.

after this restarting of consul server containers resolved the issue.
Can you please suggest, how to overcome this issue in future? and why the consul agents and servers are not responding after restart?

Thanks,
Rajesh K

2

Hi @rajesh_kottakota,

Welcome to the forums.

In a Consul cluster with 3 nodes, you should have at least 2 nodes to form a quorum of nodes. In your case, when you stopped consul1 and consul2 as part of step 1, you will ideally lose the leader and the cluster will be unusable.

In your test scenario, I wonder how you retained a leader when 2 servers were stopped out of 3. I would encourage you to revisit the tests.