I am getting following error in all 3 nodes when they are getting provisioned and then trying to create a new cluster of 3 nodes by using auto discovery method in AWS . Telnet is working between all instances .
This is a new cluster so no previous nodes neither data exist and all the nodes are new. I am using version 1.10.4. The following commands run through Ansible when all the nodes are getting provisioned .
/opt/consul/bin/run-consul --server --cluster-tag-key "k8s-consul-servers" --cluster-tag-value "auto-join"
[WARN] agent: "autopilot.redundancy_zone_tag" is a Consul Enterprise configuration and will have no effect
[WARN] agent: "autopilot.disable_upgrade_migration" is a Consul Enterprise configuration and will have no effect
[WARN] agent: bootstrap_expect > 0: expecting 3 servers
[WARN] agent.auto_config: "autopilot.redundancy_zone_tag" is a Consul Enterprise configuration and will have no effect
[WARN] agent.auto_config: "autopilot.disable_upgrade_migration" is a Consul Enterprise configuration and will have no effect
[WARN] agent.auto_config: bootstrap_expect > 0: expecting 3 servers
[INFO] agent.server.raft: initial configuration: index=0 servers=[]
[INFO] agent.server.raft: entering follower state: follower="Node at 10.31.4.9:8300 [Follower]" leader=
[INFO] agent.server.serf.wan: serf: EventMemberJoin: i-00b2b8b4af3.us-east-1 10.31.4.9
[WARN] agent.server.serf.wan: serf: Failed to re-join any previously known node
[INFO] agent.server.serf.lan: serf: EventMemberJoin: i-00b2b8b4af3 10.31.4.9
[INFO] agent.router: Initializing LAN area manager
[WARN] agent.server.serf.lan: serf: Failed to re-join any previously known node
[INFO] agent.server: Adding LAN server: server="i-00b2b8b4af3 (Addr: tcp/10.31.4.9:8300) (DC: us-east-1)"
[INFO] agent: Started DNS server: address=0.0.0.0:8600 network=udp
[WARN] agent: grpc: addrConn.createTransport failed to connect to {us-east-1-10.31.4.9:8300 0 i-00b2b8b4af3 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.31.4.9:0->10.31.4.9:8300: operation was canceled". Reconnecting...
[INFO] agent.server: Handled event for server in area: event=member-join server=i-00b2b8b4af3.us-east-1 area=wan
[WARN] agent: grpc: addrConn.createTransport failed to connect to {us-east-1-10.31.4.9:8300 0 i-00b2b8b4af3.us-east-1 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 10.31.4.9:0->10.31.4.9:8300: operation was canceled". Reconnecting...
[INFO] agent: Started DNS server: address=0.0.0.0:8600 network=tcp
[INFO] agent: Starting server: address=[::]:8500 network=tcp protocol=http
[INFO] agent: started state syncer
[INFO] agent: Consul agent running!
[WARN] agent.server.raft: no known peers, aborting election
[ERROR] agent.anti_entropy: failed to sync remote state: error="No cluster leader"
[ERROR] agent: Coordinate update error: error="No cluster leader"