I have deployed a consul cluster with 3 nodes. i have setup --bootstrap-expect to 3. After deployment leader is elected and things works fine. However if i do a new deployment with some changes, using Terraform, aws ecs task. Following things are happening.
- 3 new tasks are getting created. These 3 new nodes will join the cluster.
- The old 3 nodes will take some time to get stopped. Till then i can see there are 6 ips in the peers API.
- Once the previous 3 nodes are stopped including the leader the election will start but consul cluster is not able to elect a leader.
I can see the following logs might be useful.
lost leadership because received a requestVote with a newer term
Also i can see the term value is diff in the 3 nodes… how it went out of sync?
The election is starting as soon as the old leader is stopped. but for some reason the leader is not able to elect.
How to fix this? is there something wrong in deployment? what is the best practice to remove the old cluster and bring back the new cluster with leader?