Consul leader election on preferred consul server

Consul
1

Hi Team,

Problem Statement:
We would like to know if there is any possible procedure for forcing particular consul server to be elected as Leader.

Description:
Our Deployment environment is configured with 5 consul servers running on different VMs inside docker containers(all in default DC). Leader will be elected on any of these consul servers during initial startup or during failover scenarios.

We have observed that on VMs where consul leader is present has bit of high CPU usage compared to others. So would like to force leader election to happen on preferred consul server which is running on non-service impacting VMs.

Verified the consul documentation, unfortunately could not able to find any procedure for forcing particular consul server to be elected as Leader.

Consul Server: Consul v1.5.3

Please let us know your comments. Also let us know if there are additional queries to understand the problem.

Thanks,
Anil kumar

2

Hi Anil,

It makes sense that the leader has a higher resource footprint compared to the non-leader instances due to the additional responsibilities it has. It sounds like you are saying that the Consul server instances could potentially boot up in a VM that is hosting services/applications.

Would it not be easier to create separate dedicated instances for Consul? That’s the recommended approach.

3

Hi Karl,

Thanks for the response.

Yes, we have consul servers running on some of the VMs which are having some critical applications.

Unfortunately because of large scale deployment, we need more consul servers and are left with having consul servers on some of critical VMs also.
So looking for a plan, if suppose consul leader is elected from these critical VMs, is there any way to force the leader to partiular consul server which is running on some dedicated OAM VMs.

Alternatives in hand are,

  1. Graceful stop of consul servers on critical VMs, to force re-election and ensure leader elected on preferred ones.
    (cons, need multiple shutdown of consul server. And also it forces multiple re-election)

  2. Need to create an dedicated VM instances, and move consul servers from critical VMs to these dedicated VMs. (need to look at customer env for available resources)

But it will be good, if we have approach from consul to failover to preferred as needed (or)
some priorty based preference which does automatic pre-emption based on priorty.

Thanks,
Anil

4

The best option is really #2, but I understand there are challenges associated with that. Any other alternative will quickly turn into a “hack” and would most likely increase complexity. Another option would be to start 1 Consul server and bootstrap it to expect 1 server. Once that server is up, start the others. But again, you can already see the new challenge this option would introduce.