kubectl logs consul-server-0 -n consul ==> Starting Consul agent... Version: '1.10.0' Node ID: 'e8a97a07-999c-3a61-7253-b06fc19f8f75' Node name: 'consul-server-0' Datacenter: 'xxxx-k8s' (Segment: '') Server: true (Bootstrap: false) Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: -1, DNS: 8600) Cluster Addr: 10.254.3.69 (LAN: 8301, WAN: 8302) Encrypt: Gossip: true, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false ==> Log data will now stream in as it occurs: 2021-11-09T19:32:31.508Z [WARN] agent: bootstrap_expect > 0: expecting 3 servers 2021-11-09T19:32:31.802Z [WARN] agent.auto_config: bootstrap_expect > 0: expecting 3 servers kubectl logs consul-server-2 -n consul ==> Starting Consul agent... Version: '1.10.0' Node ID: 'e8a97a07-999c-3a61-7253-b06fc19f8f75' Node name: 'consul-server-2' Datacenter: 'xxxx-k8s' (Segment: '') Server: true (Bootstrap: false) Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: -1, DNS: 8600) Cluster Addr: 10.254.4.63 (LAN: 8301, WAN: 8302) Encrypt: Gossip: true, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false ==> Log data will now stream in as it occurs: 2021-11-09T19:57:56.938Z [WARN] agent: bootstrap_expect > 0: expecting 3 servers 2021-11-09T19:57:57.140Z [WARN] agent.auto_config: bootstrap_expect > 0: expecting 3 servers kubectl logs consul-server-1 -n consul ==> Starting Consul agent... Version: '1.10.0' Node ID: 'e8a97a07-999c-3a61-7253-b06fc19f8f75' Node name: 'consul-server-1' Datacenter: 'xxx-k8s' (Segment: '') Server: true (Bootstrap: false) Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: -1, DNS: 8600) Cluster Addr: 10.254.1.71 (LAN: 8301, WAN: 8302) Encrypt: Gossip: true, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false ==> Log data will now stream in as it occurs: 2021-11-09T19:24:31.938Z [WARN] agent: bootstrap_expect > 0: expecting 3 servers 2021-11-09T19:24:32.042Z [WARN] agent.auto_config: bootstrap_expect > 0: expecting 3 servers 2021-11-09T19:45:34.766Z [INFO] agent.server.raft: initial configuration: index=5431 servers="[{Suffrage:Voter ID:e8a97a07-999c-3a61-7253-b06fc19f8f75 Address:10.254.4.59:8300}]" 2021-11-09T19:45:34.766Z [INFO] agent.server.raft: entering follower state: follower="Node at 10.254.1.71:8300 [Follower]" leader= 2021-11-09T19:45:34.837Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul-server-1.waterloo-k8s 10.254.1.71 2021-11-09T19:45:34.837Z [INFO] agent.server.serf.wan: serf: Attempting re-join to previously known node: consul-server-2.waterloo-k8s: 10.254.4.59:8302 2021-11-09T19:45:34.839Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-server-1 10.254.1.71 2021-11-09T19:45:34.839Z [INFO] agent.router: Initializing LAN area manager 2021-11-09T19:45:34.839Z [INFO] agent.server.serf.lan: serf: Attempting re-join to previously known node: consul-server-2: 10.254.4.59:8301 2021-11-09T19:45:34.839Z [WARN] agent: grpc: addrConn.createTransport failed to connect to {10.254.1.71:8300 0 consul-server-1 }. Err :connection error: desc = "transport: Error while dialing dial tcp 10.254.1.71:8300: operation was canceled". Reconnecting... 2021-11-09T19:45:34.839Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul-server-1.waterloo-k8s area=wan 2021-11-09T19:45:34.839Z [INFO] agent.server: Adding LAN server: server="consul-server-1 (Addr: tcp/10.254.1.71:8300) (DC: waterloo-k8s)" 2021-11-09T19:45:34.840Z [INFO] agent.server: Raft data found, disabling bootstrap mode 2021-11-09T19:45:34.844Z [INFO] agent: Started DNS server: address=0.0.0.0:8600 network=tcp 2021-11-09T19:45:34.844Z [INFO] agent: Started DNS server: address=0.0.0.0:8600 network=udp 2021-11-09T19:45:34.936Z [INFO] agent: Starting server: address=[::]:8500 network=tcp protocol=http 2021-11-09T19:45:34.936Z [WARN] agent: DEPRECATED Backwards compatibility with pre-1.9 metrics enabled. These metrics will be removed in a future version of Consul. Set `telemetry { disable_compat_1.9 = true }` to disable them. 2021-11-09T19:45:34.936Z [INFO] agent: started state syncer 2021-11-09T19:45:34.936Z [INFO] agent: Consul agent running! 2021-11-09T19:45:34.937Z [INFO] agent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws azure digitalocean gce k8s linode mdns os packet scaleway softlayer tencentcloud triton vsphere" 2021-11-09T19:45:34.937Z [INFO] agent: Joining cluster...: cluster=LAN 2021-11-09T19:45:34.937Z [INFO] agent: (LAN) joining: lan_addresses=[consul-server-0.consul-server.consul.svc:8301, consul-server-1.consul-server.consul.svc:8301, consul-server-2.consul-server.consul.svc:8301] 2021-11-09T19:45:35.597Z [WARN] agent.server.memberlist.lan: memberlist: Failed to resolve consul-server-2.consul-server.consul.svc:8301: lookup consul-server-2.consul-server.consul.svc on 10.254.128.10:53: no such host 2021-11-09T19:45:35.597Z [INFO] agent: (LAN) joined: number_of_nodes=1 2021-11-09T19:45:35.597Z [INFO] agent: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=1 2021-11-09T19:45:40.001Z [WARN] agent.server.raft: heartbeat timeout reached, starting election: last-leader= 2021-11-09T19:45:40.002Z [INFO] agent.server.raft: entering candidate state: node="Node at 10.254.1.71:8300 [Candidate]" term=15 2021-11-09T19:45:40.064Z [INFO] agent.server.raft: election won: tally=1 2021-11-09T19:45:40.064Z [INFO] agent.server.raft: entering leader state: leader="Node at 10.254.1.71:8300 [Leader]" 2021-11-09T19:45:40.064Z [INFO] agent.server: cluster leadership acquired 2021-11-09T19:45:40.064Z [INFO] agent.server: New leader elected: payload=consul-server-1 2021-11-09T19:45:44.935Z [WARN] agent.server.serf.lan: serf: Failed to re-join any previously known node 2021-11-09T19:45:53.035Z [WARN] agent.server.serf.wan: serf: Failed to re-join any previously known node 2021-11-09T19:45:53.136Z [INFO] agent: Synced node info 2021-11-09T19:45:53.137Z [ERROR] agent.server.autopilot: Error when computing next state: error="cannot detect the current leader server id from its address: 10.254.1.71:8300" 2021-11-09T19:45:53.137Z [INFO] agent.leader: started routine: routine="federation state anti-entropy" 2021-11-09T19:45:53.137Z [INFO] agent.leader: started routine: routine="federation state pruning" 2021-11-09T19:45:53.137Z [INFO] agent.leader: started routine: routine="intermediate cert renew watch" 2021-11-09T19:45:53.137Z [INFO] agent.leader: started routine: routine="CA root pruning" 2021-11-09T19:45:53.137Z [INFO] agent.server.raft: updating configuration: command=AddStaging server-id=e8a97a07-999c-3a61-7253-b06fc19f8f75 server-addr=10.254.1.71:8300 servers="[{Suffrage:Voter ID:e8a97a07-999c-3a61-7253-b06fc19f8f75 Address:10.254.1.71:8300}]" 2021-11-09T19:45:53.243Z [INFO] agent.server: member joined, marking health alive: member=consul-server-1 2021-11-09T19:46:03.370Z [INFO] agent: Newer Consul version available: new_version=1.10.3 current_version=1.10.0