==> Starting Consul agent... Version: '1.9.3' Node ID: '0bd86151-612d-47e6-a5c7-08287c3515e5' Node name: 'consul-eap-ip-10-27-54-188-ip-10-27-54-188.us-west-2.compute.internal' Datacenter: 'apiq-dev-us1-wei-eap' (Segment: '') Server: true (Bootstrap: false) Client Addr: [10.27.54.188] (HTTP: 8500, HTTPS: 8501, gRPC: 8502, DNS: 8600) Cluster Addr: 10.27.54.188 (LAN: 8301, WAN: 8302) Encrypt: Gossip: false, TLS-Outgoing: true, TLS-Incoming: false, Auto-Encrypt-TLS: false ==> Log data will now stream in as it occurs: 2021-04-16T16:16:54.930Z [WARN] agent: skipping file local/config/consul-server.crt, extension must be .hcl or .json, or config format must be set 2021-04-16T16:16:54.930Z [WARN] agent: skipping file local/config/consul-server.key, extension must be .hcl or .json, or config format must be set 2021-04-16T16:16:54.930Z [WARN] agent: skipping file local/config/envca.crt, extension must be .hcl or .json, or config format must be set 2021-04-16T16:16:54.930Z [WARN] agent: Node name "consul-eap-ip-10-27-54-188-ip-10-27-54-188.us-west-2.compute.internal" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. 2021-04-16T16:16:54.930Z [WARN] agent: bootstrap_expect > 0: expecting 3 servers 2021-04-16T16:16:54.959Z [WARN] agent.auto_config: skipping file local/config/consul-server.crt, extension must be .hcl or .json, or config format must be set 2021-04-16T16:16:54.965Z [WARN] agent.auto_config: skipping file local/config/consul-server.key, extension must be .hcl or .json, or config format must be set 2021-04-16T16:16:54.965Z [WARN] agent.auto_config: skipping file local/config/envca.crt, extension must be .hcl or .json, or config format must be set 2021-04-16T16:16:54.965Z [WARN] agent.auto_config: Node name "consul-eap-ip-10-27-54-188-ip-10-27-54-188.us-west-2.compute.internal" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. 2021-04-16T16:16:54.965Z [WARN] agent.auto_config: bootstrap_expect > 0: expecting 3 servers 2021-04-16T16:16:55.004Z [INFO] agent.server.raft: restored from snapshot: id=2-229447-1618519891672 2021-04-16T16:16:58.942Z [INFO] agent.server.raft: initial configuration: index=245040 servers="[{Suffrage:Voter ID:659cafd8-5944-ecc0-39f4-9639d6f91395 Address:10.27.52.42:8300} {Suffrage:Voter ID:607ffdfd-64be-90b8-2ee1-f65c801aef7a Address:10.27.53.99:8300} {Suffrage:Voter ID:0bd86151-612d-47e6-a5c7-08287c3515e5 Address:10.27.54.17:8300}]" 2021-04-16T16:16:58.942Z [INFO] agent.server.raft: entering follower state: follower="Node at 10.27.54.188:8300 [Follower]" leader= 2021-04-16T16:16:58.944Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul-eap-ip-10-27-54-188-ip-10-27-54-188.us-west-2.compute.internal.apiq-dev-us1-wei-eap 10.27.54.188 2021-04-16T16:16:58.944Z [INFO] agent.server.serf.wan: serf: Attempting re-join to previously known node: consul-eap-ip-10-27-53-99-ip-10-27-53-99.us-west-2.compute.internal.apiq-dev-us1-wei-eap: 10.27.53.99:8302 2021-04-16T16:16:58.945Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-eap-ip-10-27-54-188-ip-10-27-54-188.us-west-2.compute.internal 10.27.54.188 2021-04-16T16:16:58.945Z [INFO] agent.router: Initializing LAN area manager 2021-04-16T16:16:58.945Z [INFO] agent.server.serf.lan: serf: Attempting re-join to previously known node: zookeeper-eap-ip-10-27-52-129: 10.27.52.129:8301 2021-04-16T16:16:58.946Z [INFO] agent: Started DNS server: address=10.27.54.188:8600 network=udp 2021-04-16T16:16:58.946Z [INFO] agent.server: Adding LAN server: server="consul-eap-ip-10-27-54-188-ip-10-27-54-188.us-west-2.compute.internal (Addr: tcp/10.27.54.188:8300) (DC: apiq-dev-us1-wei-eap)" 2021-04-16T16:16:58.946Z [INFO] agent.server: Raft data found, disabling bootstrap mode 2021-04-16T16:16:58.946Z [INFO] agent: Started DNS server: address=10.27.54.188:8600 network=tcp 2021-04-16T16:16:58.946Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul-eap-ip-10-27-54-188-ip-10-27-54-188.us-west-2.compute.internal.apiq-dev-us1-wei-eap area=wan 2021-04-16T16:16:58.947Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul-eap-ip-10-27-52-42-ip-10-27-52-42.us-west-2.compute.internal.apiq-dev-us1-wei-eap 10.27.52.42 2021-04-16T16:16:58.947Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul-eap-ip-10-27-53-99-ip-10-27-53-99.us-west-2.compute.internal.apiq-dev-us1-wei-eap 10.27.53.99 2021-04-16T16:16:58.947Z [INFO] agent.server.serf.wan: serf: Re-joined to previously known node: consul-eap-ip-10-27-53-99-ip-10-27-53-99.us-west-2.compute.internal.apiq-dev-us1-wei-eap: 10.27.53.99:8302 2021-04-16T16:16:58.947Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul-eap-ip-10-27-52-42-ip-10-27-52-42.us-west-2.compute.internal.apiq-dev-us1-wei-eap area=wan 2021-04-16T16:16:58.947Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul-eap-ip-10-27-53-99-ip-10-27-53-99.us-west-2.compute.internal.apiq-dev-us1-wei-eap area=wan 2021-04-16T16:16:58.947Z [INFO] agent: Starting server: address=10.27.54.188:8500 network=tcp protocol=http 2021-04-16T16:16:58.947Z [INFO] agent: Starting server: address=[::]:8501 network=tcp protocol=https 2021-04-16T16:16:58.947Z [WARN] agent: DEPRECATED Backwards compatibility with pre-1.9 metrics enabled. These metrics will be removed in a future version of Consul. Set `telemetry { disable_compat_1.9 = true }` to disable them. 2021-04-16T16:16:58.948Z [INFO] agent: started state syncer ==> Consul agent running! 2021-04-16T16:16:58.948Z [INFO] agent: Started gRPC server: address=10.27.54.188:8502 network=tcp 2021-04-16T16:16:58.948Z [INFO] agent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws azure digitalocean gce k8s linode mdns os packet scaleway softlayer tencentcloud triton vsphere" 2021-04-16T16:16:58.948Z [INFO] agent: Joining cluster...: cluster=LAN 2021-04-16T16:16:58.948Z [INFO] agent: discover-aws: Address type is not supported. Valid values are {private_v4,public_v4,public_v6}. Falling back to 'private_v4': cluster=LAN 2021-04-16T16:16:58.948Z [INFO] agent: discover-aws: Region not provided. Looking up region in metadata...: cluster=LAN 2021-04-16T16:16:58.956Z [INFO] agent: discover-aws: Region is us-west-2: cluster=LAN 2021-04-16T16:16:58.956Z [INFO] agent: discover-aws: Filter instances with ReplicaResourceId=consul-eap-wei: cluster=LAN 2021-04-16T16:16:59.164Z [INFO] agent: discover-aws: Instance i-02b3aa0cb36cdb934 has private ip 10.27.54.188: cluster=LAN 2021-04-16T16:16:59.164Z [INFO] agent: discover-aws: Instance i-067e6336c90a21d84 has private ip 10.27.52.42: cluster=LAN 2021-04-16T16:16:59.165Z [INFO] agent: discover-aws: Instance i-0c134c6fe4fb0bae6 has private ip 10.27.53.99: cluster=LAN 2021-04-16T16:16:59.165Z [INFO] agent: Discovered servers: cluster=LAN cluster=LAN servers="10.27.54.188 10.27.52.42 10.27.53.99" 2021-04-16T16:16:59.165Z [INFO] agent: (LAN) joining: lan_addresses=[10.27.54.188, 10.27.52.42, 10.27.53.99] 2021-04-16T16:16:59.168Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: zookeeper-eap-ip-10-27-54-142 10.27.54.142 2021-04-16T16:16:59.168Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: kafka-eap-ip-10-27-53-177 10.27.53.177 2021-04-16T16:16:59.168Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: util-eap-ip-10-27-53-71 10.27.53.71 2021-04-16T16:16:59.169Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: kafka-eap-ip-10-27-54-238 10.27.54.238 2021-04-16T16:16:59.169Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: util-eap-ip-10-27-52-74 10.27.52.74 2021-04-16T16:16:59.169Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: iqos-eap-ip-10-27-52-40 10.27.52.40 2021-04-16T16:16:59.169Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: zookeeper-eap-ip-10-27-53-111 10.27.53.111 2021-04-16T16:16:59.169Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: iqos-eap-ip-10-27-54-55 10.27.54.55 2021-04-16T16:16:59.169Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: zookeeper-eap-ip-10-27-52-129 10.27.52.129 2021-04-16T16:16:59.169Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-eap-ip-10-27-52-42-ip-10-27-52-42.us-west-2.compute.internal 10.27.52.42 2021-04-16T16:16:59.169Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: kafka-eap-ip-10-27-52-135 10.27.52.135 2021-04-16T16:16:59.169Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: util-eap-ip-10-27-54-179 10.27.54.179 2021-04-16T16:16:59.170Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-eap-ip-10-27-53-99-ip-10-27-53-99.us-west-2.compute.internal 10.27.53.99 2021-04-16T16:16:59.170Z [INFO] agent.server: Adding LAN server: server="consul-eap-ip-10-27-52-42-ip-10-27-52-42.us-west-2.compute.internal (Addr: tcp/10.27.52.42:8300) (DC: apiq-dev-us1-wei-eap)" 2021-04-16T16:16:59.170Z [INFO] agent.server: Adding LAN server: server="consul-eap-ip-10-27-53-99-ip-10-27-53-99.us-west-2.compute.internal (Addr: tcp/10.27.53.99:8300) (DC: apiq-dev-us1-wei-eap)" 2021-04-16T16:16:59.175Z [INFO] agent: (LAN) joined: number_of_nodes=3 2021-04-16T16:16:59.175Z [INFO] agent: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=3 2021-04-16T16:16:59.175Z [WARN] agent.server.raft: failed to get previous log: previous-index=245113 last-index=245048 error="log not found" 2021-04-16T16:16:59.367Z [INFO] agent: Synced node info 2021-04-16T16:17:08.946Z [INFO] agent.server.serf.lan: serf: Attempting re-join to previously known node: zookeeper-eap-ip-10-27-53-111: 10.27.53.111:8301 2021-04-16T16:17:18.946Z [INFO] agent.server.serf.lan: serf: Attempting re-join to previously known node: consul-eap-ip-10-27-53-99-ip-10-27-53-99.us-west-2.compute.internal: 10.27.53.99:8301 2021-04-16T16:17:18.950Z [INFO] agent.server.serf.lan: serf: Re-joined to previously known node: consul-eap-ip-10-27-53-99-ip-10-27-53-99.us-west-2.compute.internal: 10.27.53.99:8301 2021-04-16T16:17:28.032Z [INFO] agent: Newer Consul version available: new_version=1.9.4 current_version=1.9.3 2021-04-16T16:17:50.070Z [WARN] agent.server.raft: heartbeat timeout reached, starting election: last-leader=10.27.52.42:8300 2021-04-16T16:17:50.070Z [INFO] agent.server.raft: entering candidate state: node="Node at 10.27.54.188:8300 [Candidate]" term=41 2021-04-16T16:17:56.509Z [WARN] agent.server.raft: Election timeout reached, restarting election 2021-04-16T16:17:56.509Z [INFO] agent.server.raft: entering candidate state: node="Node at 10.27.54.188:8300 [Candidate]" term=42 2021-04-16T16:18:06.007Z [WARN] agent.server.raft: Election timeout reached, restarting election 2021-04-16T16:18:06.007Z [INFO] agent.server.raft: entering candidate state: node="Node at 10.27.54.188:8300 [Candidate]" term=43 2021-04-16T16:18:08.188Z [ERROR] agent: Coordinate update error: error="No cluster leader" 2021-04-16T16:18:13.284Z [WARN] agent.server.raft: Election timeout reached, restarting election 2021-04-16T16:18:13.284Z [INFO] agent.server.raft: entering candidate state: node="Node at 10.27.54.188:8300 [Candidate]" term=44 2021-04-16T16:18:22.608Z [WARN] agent.server.raft: Election timeout reached, restarting election 2021-04-16T16:18:22.609Z [INFO] agent.server.raft: entering candidate state: node="Node at 10.27.54.188:8300 [Candidate]" term=45 2021-04-16T16:18:28.204Z [WARN] agent.server.