No cluster leader error

Consul
1

Hello, I’ve been unable to resolve a consul server issue. The default installation via Helm works fine, but after applying my values file, I get the error “No cluster leader”.
These are the same values that I’ve used in other clusters without any issues…

From the log file I see this:

2020-10-29T18:19:38.620278077Z ==> Starting Consul agent…
2020-10-29T18:19:38.620348636Z Version: ‘1.8.4’
2020-10-29T18:19:38.620357756Z Node ID: ‘cb3d8e72-a952-e6bd-6598-7e9deb005c2d’
2020-10-29T18:19:38.620364154Z Node name: ‘consul-server-2’
2020-10-29T18:19:38.620370249Z Datacenter: ‘prod-dc1’ (Segment: ‘’)
2020-10-29T18:19:38.620402681Z Server: true (Bootstrap: false)
2020-10-29T18:19:38.620410482Z Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: -1, DNS: 8600)
2020-10-29T18:19:38.620417568Z Cluster Addr: 13.42.73.115 (LAN: 8301, WAN: 8302)
2020-10-29T18:19:38.620423612Z Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
2020-10-29T18:19:38.620429739Z
2020-10-29T18:19:38.620435171Z ==> Log data will now stream in as it occurs:
2020-10-29T18:19:38.620441312Z
2020-10-29T18:19:38.620447220Z 2020-10-29T18:19:38.619Z [WARN] agent: bootstrap_expect > 0: expecting 3 servers
2020-10-29T18:19:38.628261378Z 2020-10-29T18:19:38.628Z [WARN] agent.auto_config: bootstrap_expect > 0: expecting 3 servers
2020-10-29T18:19:38.932816952Z 2020-10-29T18:19:38.932Z [INFO] agent.server.raft: initial configuration: index=557 servers="[{Suffrage:Voter ID:4504f6b4-8cd4-3ded-fee5-acf4bc11cb8a Address:13.42.80.239:8300} {Suffrage:Voter ID:63fc1929-d2bf-95a6-960a-6e0a690a7406 Address:13.42.49.105:8300}]"
2020-10-29T18:19:38.933033279Z 2020-10-29T18:19:38.932Z [INFO] agent.server.raft: entering follower state: follower=“Node at 13.42.73.115:8300 [Follower]” leader=
2020-10-29T18:19:38.933522425Z 2020-10-29T18:19:38.933Z [WARN] agent.server.memberlist.wan: memberlist: Binding to public address without encryption!
2020-10-29T18:19:38.933695142Z 2020-10-29T18:19:38.933Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul-server-2.prod-dc1 13.42.73.115
2020-10-29T18:19:38.934388417Z 2020-10-29T18:19:38.934Z [WARN] agent.server.memberlist.lan: memberlist: Binding to public address without encryption!
2020-10-29T18:19:38.934569287Z 2020-10-29T18:19:38.934Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-server-2 13.42.73.115
2020-10-29T18:19:38.934580729Z 2020-10-29T18:19:38.934Z [INFO] agent.router: Initializing LAN area manager
2020-10-29T18:19:38.934665312Z 2020-10-29T18:19:38.934Z [INFO] agent.server: Adding LAN server: server=“consul-server-2 (Addr: tcp/13.42.73.115:8300) (DC: prod-dc1)”
2020-10-29T18:19:38.934692474Z 2020-10-29T18:19:38.934Z [INFO] agent.server: Raft data found, disabling bootstrap mode
2020-10-29T18:19:38.934761948Z 2020-10-29T18:19:38.934Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul-server-2.prod-dc1 area=wan
2020-10-29T18:19:38.934844867Z 2020-10-29T18:19:38.934Z [INFO] agent: Started DNS server: address=0.0.0.0:8600 network=udp
2020-10-29T18:19:38.934978731Z 2020-10-29T18:19:38.934Z [INFO] agent: Started DNS server: address=0.0.0.0:8600 network=tcp
2020-10-29T18:19:38.935408969Z 2020-10-29T18:19:38.935Z [INFO] agent: Started HTTP server: address=[::]:8500 network=tcp
2020-10-29T18:19:38.935565915Z 2020-10-29T18:19:38.935Z [INFO] agent: started state syncer
2020-10-29T18:19:38.935582406Z ==> Consul agent running!

Then the error starts and I’ve been unable to resolve this:

2020-10-29T18:23:50.862303090Z 2020-10-29T18:23:50.862Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4504f6b4-8cd4-3ded-fee5-acf4bc11cb8a fallback=13.42.80.239:8300 error=“Could not find address for server id 4504f6b4-8cd4-3ded-fee5-acf4bc11cb8a”
2020-10-29T18:23:50.862431806Z 2020-10-29T18:23:50.862Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=63fc1929-d2bf-95a6-960a-6e0a690a7406 fallback=13.42.49.105:8300 error=“Could not find address for server id 63fc1929-d2bf-95a6-960a-6e0a690a7406”
2020-10-29T18:23:56.032327175Z 2020-10-29T18:23:56.032Z [ERROR] agent.server.memberlist.lan: memberlist: Received invalid msgType (22) from=13.42.67.159:41114
2020-10-29T18:23:56.608431542Z 2020-10-29T18:23:56.608Z [ERROR] agent: Coordinate update error: error=“No cluster leader”
2020-10-29T18:23:59.503850356Z 2020-10-29T18:23:59.503Z [WARN] agent.server.raft: Election timeout reached, restarting election
2020-10-29T18:23:59.503889703Z 2020-10-29T18:23:59.503Z [INFO] agent.server.raft: entering candidate state: node=“Node at 13.42.73.115:8300 [Candidate]” term=425
2020-10-29T18:23:59.505250437Z 2020-10-29T18:23:59.505Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=63fc1929-d2bf-95a6-960a-6e0a690a7406 fallback=13.42.49.105:8300 error=“Could not find address for server id 63fc1929-d2bf-95a6-960a-6e0a690a7406”
2020-10-29T18:23:59.505300666Z 2020-10-29T18:23:59.505Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4504f6b4-8cd4-3ded-fee5-acf4bc11cb8a fallback=13.42.80.239:8300 error=“Could not find address for server id 4504f6b4-8cd4-3ded-fee5-acf4bc11cb8a”
2020-10-29T18:24:09.165328403Z 2020-10-29T18:24:09.165Z [WARN] agent.server.raft: Election timeout reached, restarting election
2020-10-29T18:24:09.165376395Z 2020-10-29T18:24:09.165Z [INFO] agent.server.raft: entering candidate state: node=“Node at 13.42.73.115:8300 [Candidate]” term=426
2020-10-29T18:24:09.166714369Z 2020-10-29T18:24:09.166Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=4504f6b4-8cd4-3ded-fee5-acf4bc11cb8a fallback=13.42.80.239:8300 error=“Could not find address for server id 4504f6b4-8cd4-3ded-fee5-acf4bc11cb8a”
2020-10-29T18:24:09.166855669Z 2020-10-29T18:24:09.166Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=63fc1929-d2bf-95a6-960a-6e0a690a7406 fallback=13.42.49.105:8300 error=“Could not find address for server id 63fc1929-d2bf-95a6-960a-6e0a690a7406”

2

Hi, is it possible you’ve installed and then deleted and reinstalled consul?

You need to ensure you delete the pvc’s in-between install, e.g.

k get pvc -l app=consul
NAME                           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-default-consul-server-0   Bound    pvc-fc5c4e9d-1a17-11eb-b391-42010a800020   10Gi       RWO            standard       26h


k delete pvc -l app=consul
1 Like
3

Hello,

Thanks for this much appreciated. Everything is running correctly now.

Regards…

4

what if i need the consul data back, deleting pv will also delete the old data, how to get the previous data back then

5

Thanks, this thread is really old but I have a followup question. If we are running in Kubernetes, is there a particular configuration we can use to force it to use autodiscover/rediscover (or basically, ignore the pvc and rediscover from scratch) every time it is deployed/created?

6

Hello, may I ask if the information printed out when the Consul cluster changes from three to two cannot be pushed out? How can I set the main to be pushed out