gonna sleep for 1 seconds before starting to avoid too-fast restarts auth'ing to vault cluster: https://vault.wei.us1.dev.riq.com Done generate certificates for the server ==> Starting Consul agent... Version: '1.9.3' Node ID: '607ffdfd-64be-90b8-2ee1-f65c801aef7a' Node name: 'consul-eap-ip-10-27-53-99-ip-10-27-53-99.us-west-2.compute.internal' Datacenter: 'apiq-dev-us1-wei-eap' (Segment: '') Server: true (Bootstrap: false) Client Addr: [10.27.53.99] (HTTP: 8500, HTTPS: 8501, gRPC: 8502, DNS: 8600) Cluster Addr: 10.27.53.99 (LAN: 8301, WAN: 8302) Encrypt: Gossip: false, TLS-Outgoing: true, TLS-Incoming: false, Auto-Encrypt-TLS: false ==> Log data will now stream in as it occurs: 2021-04-17T01:26:18.491Z [WARN] agent: skipping file local/config/consul-server.crt, extension must be .hcl or .json, or config format must be set 2021-04-17T01:26:18.491Z [WARN] agent: skipping file local/config/consul-server.key, extension must be .hcl or .json, or config format must be set 2021-04-17T01:26:18.491Z [WARN] agent: skipping file local/config/envca.crt, extension must be .hcl or .json, or config format must be set 2021-04-17T01:26:18.491Z [WARN] agent: Node name "consul-eap-ip-10-27-53-99-ip-10-27-53-99.us-west-2.compute.internal" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. 2021-04-17T01:26:18.491Z [WARN] agent: bootstrap_expect > 0: expecting 3 servers 2021-04-17T01:26:18.512Z [WARN] agent.auto_config: skipping file local/config/consul-server.crt, extension must be .hcl or .json, or config format must be set 2021-04-17T01:26:18.512Z [WARN] agent.auto_config: skipping file local/config/consul-server.key, extension must be .hcl or .json, or config format must be set 2021-04-17T01:26:18.512Z [WARN] agent.auto_config: skipping file local/config/envca.crt, extension must be .hcl or .json, or config format must be set 2021-04-17T01:26:18.512Z [WARN] agent.auto_config: Node name "consul-eap-ip-10-27-53-99-ip-10-27-53-99.us-west-2.compute.internal" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. 2021-04-17T01:26:18.512Z [WARN] agent.auto_config: bootstrap_expect > 0: expecting 3 servers 2021-04-17T01:26:18.537Z [INFO] agent.server.raft: restored from snapshot: id=266-245833-1618592204886 2021-04-17T01:26:18.654Z [INFO] agent.server.raft: initial configuration: index=255636 servers="[{Suffrage:Voter ID:659cafd8-5944-ecc0-39f4-9639d6f91395 Address:10.27.52.42:8300} {Suffrage:Voter ID:607ffdfd-64be-90b8-2ee1-f65c801aef7a Address:10.27.53.99:8300} {Suffrage:Voter ID:0bd86151-612d-47e6-a5c7-08287c3515e5 Address:10.27.54.188:8300}]" 2021-04-17T01:26:18.655Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul-eap-ip-10-27-53-99-ip-10-27-53-99.us-west-2.compute.internal.apiq-dev-us1-wei-eap 10.27.53.99 2021-04-17T01:26:18.655Z [INFO] agent.server.raft: entering follower state: follower="Node at 10.27.53.99:8300 [Follower]" leader= 2021-04-17T01:26:18.655Z [INFO] agent.server.serf.wan: serf: Attempting re-join to previously known node: consul-eap-ip-10-27-52-42-ip-10-27-52-42.us-west-2.compute.internal.apiq-dev-us1-wei-eap: 10.27.52.42:8302 2021-04-17T01:26:18.656Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-eap-ip-10-27-53-99-ip-10-27-53-99.us-west-2.compute.internal 10.27.53.99 2021-04-17T01:26:18.656Z [INFO] agent.router: Initializing LAN area manager 2021-04-17T01:26:18.656Z [INFO] agent.server: Adding LAN server: server="consul-eap-ip-10-27-53-99-ip-10-27-53-99.us-west-2.compute.internal (Addr: tcp/10.27.53.99:8300) (DC: apiq-dev-us1-wei-eap)" 2021-04-17T01:26:18.656Z [INFO] agent.server: Raft data found, disabling bootstrap mode 2021-04-17T01:26:18.656Z [INFO] agent.server.serf.lan: serf: Attempting re-join to previously known node: zookeeper-eap-ip-10-27-52-129: 10.27.52.129:8301 2021-04-17T01:26:18.656Z [INFO] agent: Started DNS server: address=10.27.53.99:8600 network=udp 2021-04-17T01:26:18.656Z [INFO] agent: Started DNS server: address=10.27.53.99:8600 network=tcp 2021-04-17T01:26:18.658Z [INFO] agent: Starting server: address=10.27.53.99:8500 network=tcp protocol=http 2021-04-17T01:26:18.658Z [INFO] agent: Starting server: address=[::]:8501 network=tcp protocol=https 2021-04-17T01:26:18.658Z [WARN] agent: DEPRECATED Backwards compatibility with pre-1.9 metrics enabled. These metrics will be removed in a future version of Consul. Set `telemetry { disable_compat_1.9 = true }` to disable them. 2021-04-17T01:26:18.659Z [INFO] agent: started state syncer ==> Consul agent running! 2021-04-17T01:26:18.656Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul-eap-ip-10-27-53-99-ip-10-27-53-99.us-west-2.compute.internal.apiq-dev-us1-wei-eap area=wan 2021-04-17T01:26:18.660Z [INFO] agent: Started gRPC server: address=10.27.53.99:8502 network=tcp 2021-04-17T01:26:18.660Z [INFO] agent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws azure digitalocean gce k8s linode mdns os packet scaleway softlayer tencentcloud triton vsphere" 2021-04-17T01:26:18.660Z [INFO] agent: Joining cluster...: cluster=LAN 2021-04-17T01:26:18.660Z [INFO] agent: discover-aws: Address type is not supported. Valid values are {private_v4,public_v4,public_v6}. Falling back to 'private_v4': cluster=LAN 2021-04-17T01:26:18.660Z [INFO] agent: discover-aws: Region not provided. Looking up region in metadata...: cluster=LAN 2021-04-17T01:26:18.660Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul-eap-ip-10-27-54-188-ip-10-27-54-188.us-west-2.compute.internal.apiq-dev-us1-wei-eap 10.27.54.188 2021-04-17T01:26:18.661Z [WARN] agent.server.memberlist.wan: memberlist: Refuting an alive message for 'consul-eap-ip-10-27-53-99-ip-10-27-53-99.us-west-2.compute.internal.apiq-dev-us1-wei-eap' (10.27.53.99:8302) meta:([255 143 165 98 117 105 108 100 174 49 46 57 46 51 58 102 53 53 100 97 57 51 48 164 112 111 114 116 164 56 51 48 48 167 117 115 101 95 116 108 115 161 49 167 115 101 103 109 101 110 116 160 162 105 100 218 0 36 54 48 55 102 102 100 102 100 45 54 52 98 101 45 57 48 98 56 45 50 101 101 49 45 102 54 53 99 56 48 49 97 101 102 55 97 163 118 115 110 161 50 168 114 97 102 116 95 118 115 110 161 51 166 101 120 112 101 99 116 161 51 165 102 116 95 102 115 161 49 165 102 116 95 115 105 161 49 167 118 115 110 95 109 105 110 161 50 164 97 99 108 115 161 48 164 114 111 108 101 166 99 111 110 115 117 108 162 100 99 180 97 112 105 113 45 100 101 118 45 117 115 49 45 119 101 105 45 101 97 112 167 118 115 110 95 109 97 120 161 51] VS [255 143 166 101 120 112 101 99 116 161 51 165 102 116 95 115 105 161 49 167 115 101 103 109 101 110 116 160 162 105 100 218 0 36 54 48 55 102 102 100 102 100 45 54 52 98 101 45 57 48 98 56 45 50 101 101 49 45 102 54 53 99 56 48 49 97 101 102 55 97 168 114 97 102 116 95 118 115 110 161 51 165 98 117 105 108 100 174 49 46 57 46 51 58 102 53 53 100 97 57 51 48 164 112 111 114 116 164 56 51 48 48 165 102 116 95 102 115 161 49 162 100 99 180 97 112 105 113 45 100 101 118 45 117 115 49 45 119 101 105 45 101 97 112 163 118 115 110 161 50 167 118 115 110 95 109 105 110 161 50 164 114 111 108 101 166 99 111 110 115 117 108 167 117 115 101 95 116 108 115 161 49 164 97 99 108 115 161 48 167 118 115 110 95 109 97 120 161 51]), vsn:([1 5 2 2 5 4] VS [1 5 2 2 5 4]) 2021-04-17T01:26:18.661Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul-eap-ip-10-27-52-42-ip-10-27-52-42.us-west-2.compute.internal.apiq-dev-us1-wei-eap 10.27.52.42 2021-04-17T01:26:18.661Z [INFO] agent.server.serf.wan: serf: Re-joined to previously known node: consul-eap-ip-10-27-52-42-ip-10-27-52-42.us-west-2.compute.internal.apiq-dev-us1-wei-eap: 10.27.52.42:8302 2021-04-17T01:26:18.661Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul-eap-ip-10-27-54-188-ip-10-27-54-188.us-west-2.compute.internal.apiq-dev-us1-wei-eap area=wan 2021-04-17T01:26:18.661Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul-eap-ip-10-27-52-42-ip-10-27-52-42.us-west-2.compute.internal.apiq-dev-us1-wei-eap area=wan 2021-04-17T01:26:18.662Z [INFO] agent: discover-aws: Region is us-west-2: cluster=LAN 2021-04-17T01:26:18.662Z [INFO] agent: discover-aws: Filter instances with ReplicaResourceId=consul-eap-wei: cluster=LAN 2021-04-17T01:26:18.875Z [INFO] agent: discover-aws: Instance i-02b3aa0cb36cdb934 has private ip 10.27.54.188: cluster=LAN 2021-04-17T01:26:18.875Z [INFO] agent: discover-aws: Instance i-067e6336c90a21d84 has private ip 10.27.52.42: cluster=LAN 2021-04-17T01:26:18.875Z [INFO] agent: discover-aws: Instance i-0c134c6fe4fb0bae6 has private ip 10.27.53.99: cluster=LAN 2021-04-17T01:26:18.875Z [INFO] agent: Discovered servers: cluster=LAN cluster=LAN servers="10.27.54.188 10.27.52.42 10.27.53.99" 2021-04-17T01:26:18.875Z [INFO] agent: (LAN) joining: lan_addresses=[10.27.54.188, 10.27.52.42, 10.27.53.99] 2021-04-17T01:26:18.880Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: zookeeper-eap-ip-10-27-54-142 10.27.54.142 2021-04-17T01:26:18.880Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: util-eap-ip-10-27-54-179 10.27.54.179 2021-04-17T01:26:18.881Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: zookeeper-eap-ip-10-27-53-111 10.27.53.111 2021-04-17T01:26:18.881Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: kafka-eap-ip-10-27-52-135 10.27.52.135 2021-04-17T01:26:18.881Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: util-eap-ip-10-27-52-74 10.27.52.74 2021-04-17T01:26:18.881Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-eap-ip-10-27-54-188-ip-10-27-54-188.us-west-2.compute.internal 10.27.54.188 2021-04-17T01:26:18.882Z [INFO] agent.server: Adding LAN server: server="consul-eap-ip-10-27-54-188-ip-10-27-54-188.us-west-2.compute.internal (Addr: tcp/10.27.54.188:8300) (DC: apiq-dev-us1-wei-eap)" 2021-04-17T01:26:18.882Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: kafka-eap-ip-10-27-53-177 10.27.53.177 2021-04-17T01:26:18.882Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-eap-ip-10-27-52-42-ip-10-27-52-42.us-west-2.compute.internal 10.27.52.42 2021-04-17T01:26:18.882Z [INFO] agent.server: Adding LAN server: server="consul-eap-ip-10-27-52-42-ip-10-27-52-42.us-west-2.compute.internal (Addr: tcp/10.27.52.42:8300) (DC: apiq-dev-us1-wei-eap)" 2021-04-17T01:26:18.882Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: util-eap-ip-10-27-53-71 10.27.53.71 2021-04-17T01:26:18.883Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: iqos-eap-ip-10-27-52-40 10.27.52.40 2021-04-17T01:26:18.883Z [WARN] agent.server.memberlist.lan: memberlist: Refuting a suspect message (from: consul-eap-ip-10-27-53-99-ip-10-27-53-99.us-west-2.compute.internal) 2021-04-17T01:26:18.883Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: iqos-eap-ip-10-27-54-55 10.27.54.55 2021-04-17T01:26:18.883Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: zookeeper-eap-ip-10-27-52-129 10.27.52.129 2021-04-17T01:26:18.883Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: kafka-eap-ip-10-27-54-238 10.27.54.238 2021-04-17T01:26:18.894Z [INFO] agent: (LAN) joined: number_of_nodes=3 2021-04-17T01:26:18.894Z [INFO] agent: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=3 2021-04-17T01:26:25.882Z [ERROR] agent.anti_entropy: failed to sync remote state: error="No cluster leader" 2021-04-17T01:26:26.458Z [WARN] agent.server.raft: heartbeat timeout reached, starting election: last-leader= 2021-04-17T01:26:26.458Z [INFO] agent.server.raft: entering candidate state: node="Node at 10.27.53.99:8300 [Candidate]" term=2817 2021-04-17T01:26:26.511Z [INFO] agent: Newer Consul version available: new_version=1.9.4 current_version=1.9.3 2021-04-17T01:26:28.656Z [INFO] agent.server.serf.lan: serf: Attempting re-join to previously known node: util-eap-ip-10-27-52-74: 10.27.52.74:8301 2021-04-17T01:26:28.662Z [INFO] agent.server.serf.lan: serf: Re-joined to previously known node: util-eap-ip-10-27-52-74: 10.27.52.74:8301 2021-04-17T01:26:30.656Z [INFO] agent.server.memberlist.lan: memberlist: Suspect consul-eap-ip-10-27-52-42-ip-10-27-52-42.us-west-2.compute.internal has failed, no acks received 2021-04-17T01:26:31.079Z [INFO] agent.server.raft: entering follower state: follower="Node at 10.27.53.99:8300 [Follower]" leader= 2021-04-17T01:26:31.170Z [INFO] agent.server: New leader elected: payload=consul-eap-ip-10-27-54-188-ip-10-27-54-188.us-west-2.compute.internal 2021-04-17T01:26:32.007Z [INFO] agent: Synced node info 2021-04-17T01:26:32.498Z [INFO] agent.server.serf.lan: serf: EventMemberUpdate: consul-eap-ip-10-27-52-42-ip-10-27-52-42.us-west-2.compute.internal 2021-04-17T01:26:32.498Z [INFO] agent.server: Updating LAN server: server="consul-eap-ip-10-27-52-42-ip-10-27-52-42.us-west-2.compute.internal (Addr: tcp/10.27.52.42:8300) (DC: apiq-dev-us1-wei-eap)" 2021-04-17T01:26:32.878Z [INFO] agent.server.serf.wan: serf: EventMemberUpdate: consul-eap-ip-10-27-52-42-ip-10-27-52-42.us-west-2.compute.internal.apiq-dev-us1-wei-eap 2021-04-17T01:26:32.878Z [INFO] agent.server: Handled event for server in area: event=member-update server=consul-eap-ip-10-27-52-42-ip-10-27-52-42.us-west-2.compute.internal.apiq-dev-us1-wei-eap area=wan 2021-04-17T01:26:33.655Z [INFO] agent.server.memberlist.wan: memberlist: Suspect consul-eap-ip-10-27-52-42-ip-10-27-52-42.us-west-2.compute.internal.apiq-dev-us1-wei-eap has failed, no acks received 2021-04-17T01:26:46.216Z [INFO] agent.server.serf.lan: serf: EventMemberUpdate: consul-eap-ip-10-27-54-188-ip-10-27-54-188.us-west-2.compute.internal 2021-04-17T01:26:46.217Z [INFO] agent.server: Updating LAN server: server="consul-eap-ip-10-27-54-188-ip-10-27-54-188.us-west-2.compute.internal (Addr: tcp/10.27.54.188:8300) (DC: apiq-dev-us1-wei-eap)" 2021-04-17T01:26:46.217Z [INFO] agent.server.serf.wan: serf: EventMemberUpdate: consul-eap-ip-10-27-54-188-ip-10-27-54-188.us-west-2.compute.internal.apiq-dev-us1-wei-eap 2021-04-17T01:26:46.217Z [INFO] agent.server: Handled event for server in area: event=member-update server=consul-eap-ip-10-27-54-188-ip-10-27-54-188.us-west-2.compute.internal.apiq-dev-us1-wei-eap area=wan 2021-04-17T01:26:49.427Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.52.42:8300 leader=10.27.54.188:8300 2021-04-17T01:26:49.657Z [INFO] agent.server: New leader elected: payload=consul-eap-ip-10-27-52-42-ip-10-27-52-42.us-west-2.compute.internal 2021-04-17T01:27:08.204Z [ERROR] agent.server.memberlist.lan: memberlist: Push/Pull with zookeeper-eap-ip-10-27-52-129 failed: dial tcp 10.27.52.129:8301: i/o timeout 2021-04-17T01:29:18.219Z [ERROR] agent.server.memberlist.lan: memberlist: Push/Pull with zookeeper-eap-ip-10-27-53-111 failed: dial tcp 10.27.53.111:8301: i/o timeout 2021-04-17T01:29:58.219Z [ERROR] agent.server.memberlist.lan: memberlist: Push/Pull with kafka-eap-ip-10-27-54-238 failed: dial tcp 10.27.54.238:8301: i/o timeout 2021-04-17T01:31:38.226Z [ERROR] agent.server.memberlist.lan: memberlist: Push/Pull with kafka-eap-ip-10-27-54-238 failed: dial tcp 10.27.54.238:8301: i/o timeout 2021-04-17T01:32:18.227Z [ERROR] agent.server.memberlist.lan: memberlist: Push/Pull with kafka-eap-ip-10-27-54-238 failed: dial tcp 10.27.54.238:8301: i/o timeout