Install Consul through latest helm chart and I am receive errors for No cluster leader

I setup a home lab with a new Kubernetes cluster for testing. I have the cluster stood up and now I am attempting to install consul onto the cluster through helm charts. I pulled the latest stable git repo per the documentation (v17) of the consul helm chart. All the pods start up and go into a running states, but fail to get to a ready state. The logs state that no cluster leader is available. Below is the logs from one of the pods.

2020/03/10 13:24:45 [ERR] agent: Coordinate update error: No cluster leader
2020/03/10 13:25:11 [ERR] agent: failed to sync remote state: No cluster leader
2020/03/10 13:25:13 [WARN] memberlist: Failed to resolve consul-consul-server-2.consul-consul-server.default.svc: lookup consul-consul-server-2.consul-consul-server.default.svc on 10.96.0.10:53: read udp 10.244.2.41:53127->10.96.0.10:53: i/o timeout
2020/03/10 13:25:13 [WARN] agent: (LAN) couldn’t join: 0 Err: 3 errors occurred:
* Failed to resolve consul-consul-server-0.consul-consul-server.default.svc: lookup consul-consul-server-0.consul-consul-server.default.svc on 10.96.0.10:53: read udp 10.244.2.41:51210->10.96.0.10:53: i/o timeout
* Failed to resolve consul-consul-server-1.consul-consul-server.default.svc: lookup consul-consul-server-1.consul-consul-server.default.svc on 10.96.0.10:53: read udp 10.244.2.41:39328->10.96.0.10:53: i/o timeout
* Failed to resolve consul-consul-server-2.consul-consul-server.default.svc: lookup consul-consul-server-2.consul-consul-server.default.svc on 10.96.0.10:53: read udp 10.244.2.41:53127->10.96.0.10:53: i/o timeout

2020/03/10 13:25:13 [WARN] agent: Join LAN failed: <nil>, retrying in 30s
2020/03/10 13:25:16 [ERR] agent: Coordinate update error: No cluster leader
2020/03/10 13:25:34 [ERR] agent: failed to sync remote state: No cluster leader
2020/03/10 13:25:43 [INFO] agent: (LAN) joining: [consul-consul-server-0.consul-consul-server.default.svc consul-consul-server-1.consul-consul-server.default.svc consul-consul-server-2.consul-consul-server.default.svc]
2020/03/10 13:25:45 [ERR] agent: Coordinate update error: No cluster leader
2020/03/10 13:26:00 [ERR] agent: failed to sync remote state: No cluster leader
2020/03/10 13:26:20 [ERR] agent: Coordinate update error: No cluster leader
2020/03/10 13:26:24 [ERR] agent: failed to sync remote state: No cluster leader
2020/03/10 13:26:25 [WARN] memberlist: Failed to resolve consul-consul-server-0.consul-consul-server.default.svc: lookup consul-consul-server-0.consul-consul-server.default.svc on 10.96.0.10:53: read udp 10.244.2.41:47117->10.96.0.10:53: i/o timeout
2020/03/10 13:26:49 [ERR] agent: Coordinate update error: No cluster leader
2020/03/10 13:26:53 [ERR] agent: failed to sync remote state: No cluster leader

Hello @Diggs27

Welcome to the Discuss forum! Thanks for posting for the first time :slight_smile: Can you post your values file, with sensitive information removed, so we can take a look at the configuration? What version of Kubernetes have you installed on the cluster, and what hardware configuration is your home cluster using?

Looking forward to hearing back from you!

Hi @jsosulska Thanks for the reply! I really appreciate you responeding.

Here is a little information about my setup.

  • 4 node cluster setup
  • Fresh Debian 10 builds with latest updates. Running on same ESXi (Esxi running on Intel NUC)
  • Kubernetes version v1.17.3
  • Docker-ce engine install
  • kubeadm used to setup the cluster

I’ve setup a nfs server and I have been using nfs-provisoner helm chart. That seems to be working and I see the PVC setup for all consul servers. For networking I was using kube-router. I tried rebuilding the cluster and installed flanner instead and I ran into the same issue. Confirmed CoreDNS is setup and resolving pods successfully.

I’m using the vanilla values.yaml from a clone of the v0.17.0 version of the git repo


I also have tried v0.16.0 without any luck. A am using helm version 3 and not version 2. I didn’t see anywhere that it wasn’t supported. Could I be using helm 2?

All nodes are healthy and all pods are in Running state as well, just not a ready state.

Hi Diggs27!

Sorry for the lag on responding to this. As you can imagine, things are a bit hectic right now for everyone.

Helm3 definitely works (up to 3.1) so that shouldn’t be an issue. Could you do me a favor and collect a few bits of information? I was hoping to see the output of kubectl get pods -A and kubectl get pvc -A. I’m curious to see how things look cluster wise (i.e. are all pods pulling down images, etc…). Another thing that would be useful is the output of a kubectl describe pod consul-consul-server-2; specifically the output messages at the end.

1 Like