gonna sleep for 1 seconds before starting to avoid too-fast restarts auth'ing to vault cluster: https://vault.wei.us1.dev.riq.com Done generate certificates for the server ==> Starting Consul agent... Version: '1.9.3' Node ID: '659cafd8-5944-ecc0-39f4-9639d6f91395' Node name: 'consul-eap-ip-10-27-52-42-ip-10-27-52-42.us-west-2.compute.internal' Datacenter: 'apiq-dev-us1-wei-eap' (Segment: '') Server: true (Bootstrap: false) Client Addr: [10.27.52.42] (HTTP: 8500, HTTPS: 8501, gRPC: 8502, DNS: 8600) Cluster Addr: 10.27.52.42 (LAN: 8301, WAN: 8302) Encrypt: Gossip: false, TLS-Outgoing: true, TLS-Incoming: false, Auto-Encrypt-TLS: false ==> Log data will now stream in as it occurs: 2021-04-17T01:26:32.293Z [WARN] agent: skipping file local/config/consul-server.crt, extension must be .hcl or .json, or config format must be set 2021-04-17T01:26:32.293Z [WARN] agent: skipping file local/config/consul-server.key, extension must be .hcl or .json, or config format must be set 2021-04-17T01:26:32.293Z [WARN] agent: skipping file local/config/envca.crt, extension must be .hcl or .json, or config format must be set 2021-04-17T01:26:32.293Z [WARN] agent: Node name "consul-eap-ip-10-27-52-42-ip-10-27-52-42.us-west-2.compute.internal" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. 2021-04-17T01:26:32.293Z [WARN] agent: bootstrap_expect > 0: expecting 3 servers 2021-04-17T01:26:32.302Z [WARN] agent.auto_config: skipping file local/config/consul-server.crt, extension must be .hcl or .json, or config format must be set 2021-04-17T01:26:32.302Z [WARN] agent.auto_config: skipping file local/config/consul-server.key, extension must be .hcl or .json, or config format must be set 2021-04-17T01:26:32.302Z [WARN] agent.auto_config: skipping file local/config/envca.crt, extension must be .hcl or .json, or config format must be set 2021-04-17T01:26:32.302Z [WARN] agent.auto_config: Node name "consul-eap-ip-10-27-52-42-ip-10-27-52-42.us-west-2.compute.internal" will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes. 2021-04-17T01:26:32.302Z [WARN] agent.auto_config: bootstrap_expect > 0: expecting 3 servers 2021-04-17T01:26:32.312Z [INFO] agent.server.raft: restored from snapshot: id=266-245832-1618592203161 2021-04-17T01:26:32.376Z [INFO] agent.server.raft: initial configuration: index=255636 servers="[{Suffrage:Voter ID:659cafd8-5944-ecc0-39f4-9639d6f91395 Address:10.27.52.42:8300} {Suffrage:Voter ID:607ffdfd-64be-90b8-2ee1-f65c801aef7a Address:10.27.53.99:8300} {Suffrage:Voter ID:0bd86151-612d-47e6-a5c7-08287c3515e5 Address:10.27.54.188:8300}]" 2021-04-17T01:26:32.376Z [INFO] agent.server.raft: entering follower state: follower="Node at 10.27.52.42:8300 [Follower]" leader= 2021-04-17T01:26:32.377Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul-eap-ip-10-27-52-42-ip-10-27-52-42.us-west-2.compute.internal.apiq-dev-us1-wei-eap 10.27.52.42 2021-04-17T01:26:32.377Z [INFO] agent.server.serf.wan: serf: Attempting re-join to previously known node: consul-eap-ip-10-27-54-188-ip-10-27-54-188.us-west-2.compute.internal.apiq-dev-us1-wei-eap: 10.27.54.188:8302 2021-04-17T01:26:32.377Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-eap-ip-10-27-52-42-ip-10-27-52-42.us-west-2.compute.internal 10.27.52.42 2021-04-17T01:26:32.378Z [INFO] agent.router: Initializing LAN area manager 2021-04-17T01:26:32.378Z [INFO] agent.server.serf.lan: serf: Attempting re-join to previously known node: kafka-eap-ip-10-27-52-135: 10.27.52.135:8301 2021-04-17T01:26:32.378Z [INFO] agent: Started DNS server: address=10.27.52.42:8600 network=udp 2021-04-17T01:26:32.378Z [INFO] agent.server: Adding LAN server: server="consul-eap-ip-10-27-52-42-ip-10-27-52-42.us-west-2.compute.internal (Addr: tcp/10.27.52.42:8300) (DC: apiq-dev-us1-wei-eap)" 2021-04-17T01:26:32.378Z [INFO] agent.server: Raft data found, disabling bootstrap mode 2021-04-17T01:26:32.378Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul-eap-ip-10-27-52-42-ip-10-27-52-42.us-west-2.compute.internal.apiq-dev-us1-wei-eap area=wan 2021-04-17T01:26:32.378Z [INFO] agent: Started DNS server: address=10.27.52.42:8600 network=tcp 2021-04-17T01:26:32.379Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul-eap-ip-10-27-53-99-ip-10-27-53-99.us-west-2.compute.internal.apiq-dev-us1-wei-eap 10.27.53.99 2021-04-17T01:26:32.379Z [WARN] agent.server.memberlist.wan: memberlist: Refuting an alive message for 'consul-eap-ip-10-27-52-42-ip-10-27-52-42.us-west-2.compute.internal.apiq-dev-us1-wei-eap' (10.27.52.42:8302) meta:([255 143 162 100 99 180 97 112 105 113 45 100 101 118 45 117 115 49 45 119 101 105 45 101 97 112 163 118 115 110 161 50 164 97 99 108 115 161 48 164 114 111 108 101 166 99 111 110 115 117 108 164 112 111 114 116 164 56 51 48 48 165 102 116 95 102 115 161 49 168 114 97 102 116 95 118 115 110 161 51 166 101 120 112 101 99 116 161 51 167 117 115 101 95 116 108 115 161 49 165 102 116 95 115 105 161 49 165 98 117 105 108 100 174 49 46 57 46 51 58 102 53 53 100 97 57 51 48 162 105 100 218 0 36 54 53 57 99 97 102 100 56 45 53 57 52 52 45 101 99 99 48 45 51 57 102 52 45 57 54 51 57 100 54 102 57 49 51 57 53 167 118 115 110 95 109 105 110 161 50 167 118 115 110 95 109 97 120 161 51 167 115 101 103 109 101 110 116 160] VS [255 143 167 115 101 103 109 101 110 116 160 167 118 115 110 95 109 97 120 161 51 165 102 116 95 102 115 161 49 162 105 100 218 0 36 54 53 57 99 97 102 100 56 45 53 57 52 52 45 101 99 99 48 45 51 57 102 52 45 57 54 51 57 100 54 102 57 49 51 57 53 163 118 115 110 161 50 167 118 115 110 95 109 105 110 161 50 165 98 117 105 108 100 174 49 46 57 46 51 58 102 53 53 100 97 57 51 48 164 112 111 114 116 164 56 51 48 48 166 101 120 112 101 99 116 161 51 167 117 115 101 95 116 108 115 161 49 165 102 116 95 115 105 161 49 164 114 111 108 101 166 99 111 110 115 117 108 162 100 99 180 97 112 105 113 45 100 101 118 45 117 115 49 45 119 101 105 45 101 97 112 168 114 97 102 116 95 118 115 110 161 51 164 97 99 108 115 161 48]), vsn:([1 5 2 2 5 4] VS [1 5 2 2 5 4]) 2021-04-17T01:26:32.379Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul-eap-ip-10-27-54-188-ip-10-27-54-188.us-west-2.compute.internal.apiq-dev-us1-wei-eap 10.27.54.188 2021-04-17T01:26:32.379Z [INFO] agent.server.serf.wan: serf: Re-joined to previously known node: consul-eap-ip-10-27-54-188-ip-10-27-54-188.us-west-2.compute.internal.apiq-dev-us1-wei-eap: 10.27.54.188:8302 2021-04-17T01:26:32.379Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul-eap-ip-10-27-53-99-ip-10-27-53-99.us-west-2.compute.internal.apiq-dev-us1-wei-eap area=wan 2021-04-17T01:26:32.379Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul-eap-ip-10-27-54-188-ip-10-27-54-188.us-west-2.compute.internal.apiq-dev-us1-wei-eap area=wan 2021-04-17T01:26:32.380Z [INFO] agent: Starting server: address=10.27.52.42:8500 network=tcp protocol=http 2021-04-17T01:26:32.380Z [INFO] agent: Starting server: address=[::]:8501 network=tcp protocol=https 2021-04-17T01:26:32.380Z [WARN] agent: DEPRECATED Backwards compatibility with pre-1.9 metrics enabled. These metrics will be removed in a future version of Consul. Set `telemetry { disable_compat_1.9 = true }` to disable them. 2021-04-17T01:26:32.380Z [INFO] agent: Started gRPC server: address=10.27.52.42:8502 network=tcp 2021-04-17T01:26:32.380Z [INFO] agent: started state syncer ==> Consul agent running! 2021-04-17T01:26:32.380Z [INFO] agent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws azure digitalocean gce k8s linode mdns os packet scaleway softlayer tencentcloud triton vsphere" 2021-04-17T01:26:32.380Z [INFO] agent: Joining cluster...: cluster=LAN 2021-04-17T01:26:32.380Z [INFO] agent: discover-aws: Address type is not supported. Valid values are {private_v4,public_v4,public_v6}. Falling back to 'private_v4': cluster=LAN 2021-04-17T01:26:32.380Z [INFO] agent: discover-aws: Region not provided. Looking up region in metadata...: cluster=LAN 2021-04-17T01:26:32.381Z [INFO] agent: discover-aws: Region is us-west-2: cluster=LAN 2021-04-17T01:26:32.381Z [INFO] agent: discover-aws: Filter instances with ReplicaResourceId=consul-eap-wei: cluster=LAN 2021-04-17T01:26:32.490Z [INFO] agent: discover-aws: Instance i-02b3aa0cb36cdb934 has private ip 10.27.54.188: cluster=LAN 2021-04-17T01:26:32.490Z [INFO] agent: discover-aws: Instance i-067e6336c90a21d84 has private ip 10.27.52.42: cluster=LAN 2021-04-17T01:26:32.490Z [INFO] agent: discover-aws: Instance i-0c134c6fe4fb0bae6 has private ip 10.27.53.99: cluster=LAN 2021-04-17T01:26:32.490Z [INFO] agent: Discovered servers: cluster=LAN cluster=LAN servers="10.27.54.188 10.27.52.42 10.27.53.99" 2021-04-17T01:26:32.490Z [INFO] agent: (LAN) joining: lan_addresses=[10.27.54.188, 10.27.52.42, 10.27.53.99] 2021-04-17T01:26:32.492Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: zookeeper-eap-ip-10-27-52-129 10.27.52.129 2021-04-17T01:26:32.492Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: util-eap-ip-10-27-53-71 10.27.53.71 2021-04-17T01:26:32.492Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: kafka-eap-ip-10-27-53-177 10.27.53.177 2021-04-17T01:26:32.493Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-eap-ip-10-27-54-188-ip-10-27-54-188.us-west-2.compute.internal 10.27.54.188 2021-04-17T01:26:32.493Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: kafka-eap-ip-10-27-52-135 10.27.52.135 2021-04-17T01:26:32.493Z [WARN] agent.server.memberlist.lan: memberlist: Refuting a suspect message (from: consul-eap-ip-10-27-52-42-ip-10-27-52-42.us-west-2.compute.internal) 2021-04-17T01:26:32.493Z [INFO] agent.server: Adding LAN server: server="consul-eap-ip-10-27-54-188-ip-10-27-54-188.us-west-2.compute.internal (Addr: tcp/10.27.54.188:8300) (DC: apiq-dev-us1-wei-eap)" 2021-04-17T01:26:32.493Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: util-eap-ip-10-27-52-74 10.27.52.74 2021-04-17T01:26:32.493Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-eap-ip-10-27-53-99-ip-10-27-53-99.us-west-2.compute.internal 10.27.53.99 2021-04-17T01:26:32.493Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: zookeeper-eap-ip-10-27-53-111 10.27.53.111 2021-04-17T01:26:32.493Z [INFO] agent.server: Adding LAN server: server="consul-eap-ip-10-27-53-99-ip-10-27-53-99.us-west-2.compute.internal (Addr: tcp/10.27.53.99:8300) (DC: apiq-dev-us1-wei-eap)" 2021-04-17T01:26:32.493Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: kafka-eap-ip-10-27-54-238 10.27.54.238 2021-04-17T01:26:32.493Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: iqos-eap-ip-10-27-52-40 10.27.52.40 2021-04-17T01:26:32.493Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: iqos-eap-ip-10-27-54-55 10.27.54.55 2021-04-17T01:26:32.494Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: zookeeper-eap-ip-10-27-54-142 10.27.54.142 2021-04-17T01:26:32.494Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: util-eap-ip-10-27-54-179 10.27.54.179 2021-04-17T01:26:32.498Z [INFO] agent: (LAN) joined: number_of_nodes=3 2021-04-17T01:26:32.498Z [INFO] agent: Join cluster completed. Synced with initial agents: cluster=LAN num_agents=3 2021-04-17T01:26:32.715Z [INFO] agent: Newer Consul version available: new_version=1.9.4 current_version=1.9.3 2021-04-17T01:26:32.839Z [INFO] agent: Synced node info 2021-04-17T01:26:42.378Z [INFO] agent.server.serf.lan: serf: Attempting re-join to previously known node: util-eap-ip-10-27-53-71: 10.27.53.71:8301 2021-04-17T01:26:42.382Z [INFO] agent.server.serf.lan: serf: Re-joined to previously known node: util-eap-ip-10-27-53-71: 10.27.53.71:8301 2021-04-17T01:26:46.211Z [INFO] agent.server.serf.lan: serf: EventMemberUpdate: consul-eap-ip-10-27-54-188-ip-10-27-54-188.us-west-2.compute.internal 2021-04-17T01:26:46.211Z [INFO] agent.server: Updating LAN server: server="consul-eap-ip-10-27-54-188-ip-10-27-54-188.us-west-2.compute.internal (Addr: tcp/10.27.54.188:8300) (DC: apiq-dev-us1-wei-eap)" 2021-04-17T01:26:46.592Z [INFO] agent.server.serf.wan: serf: EventMemberUpdate: consul-eap-ip-10-27-54-188-ip-10-27-54-188.us-west-2.compute.internal.apiq-dev-us1-wei-eap 2021-04-17T01:26:46.592Z [INFO] agent.server: Handled event for server in area: event=member-update server=consul-eap-ip-10-27-54-188-ip-10-27-54-188.us-west-2.compute.internal.apiq-dev-us1-wei-eap area=wan 2021-04-17T01:26:47.377Z [INFO] agent.server.memberlist.wan: memberlist: Suspect consul-eap-ip-10-27-54-188-ip-10-27-54-188.us-west-2.compute.internal.apiq-dev-us1-wei-eap has failed, no acks received 2021-04-17T01:26:49.417Z [WARN] agent.server.raft: heartbeat timeout reached, starting election: last-leader=10.27.54.188:8300 2021-04-17T01:26:49.417Z [INFO] agent.server.raft: entering candidate state: node="Node at 10.27.52.42:8300 [Candidate]" term=2819 2021-04-17T01:26:49.431Z [INFO] agent.server.raft: election won: tally=2 2021-04-17T01:26:49.431Z [INFO] agent.server.raft: entering leader state: leader="Node at 10.27.52.42:8300 [Leader]" 2021-04-17T01:26:49.431Z [INFO] agent.server.raft: added peer, starting replication: peer=607ffdfd-64be-90b8-2ee1-f65c801aef7a 2021-04-17T01:26:49.431Z [INFO] agent.server.raft: added peer, starting replication: peer=0bd86151-612d-47e6-a5c7-08287c3515e5 2021-04-17T01:26:49.431Z [INFO] agent.server: cluster leadership acquired 2021-04-17T01:26:49.431Z [INFO] agent.server: New leader elected: payload=consul-eap-ip-10-27-52-42-ip-10-27-52-42.us-west-2.compute.internal 2021-04-17T01:26:49.442Z [INFO] agent.server.raft: pipelining replication: peer="{Voter 607ffdfd-64be-90b8-2ee1-f65c801aef7a 10.27.53.99:8300}" 2021-04-17T01:26:49.452Z [INFO] agent.leader: started routine: routine="federation state anti-entropy" 2021-04-17T01:26:49.452Z [INFO] agent.leader: started routine: routine="federation state pruning" 2021-04-17T01:26:49.453Z [INFO] agent.leader: started routine: routine="intermediate cert renew watch" 2021-04-17T01:26:49.453Z [INFO] agent.leader: started routine: routine="CA root pruning" 2021-04-17T01:26:49.565Z [INFO] agent.server.raft: pipelining replication: peer="{Voter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.188:8300}" 2021-04-17T01:28:03.394Z [ERROR] agent.server.memberlist.lan: memberlist: Push/Pull with kafka-eap-ip-10-27-54-238 failed: dial tcp 10.27.54.238:8301: i/o timeout 2021-04-17T01:28:43.394Z [ERROR] agent.server.memberlist.lan: memberlist: Push/Pull with zookeeper-eap-ip-10-27-53-111 failed: dial tcp 10.27.53.111:8301: i/o timeout 2021-04-17T01:31:53.411Z [ERROR] agent.server.memberlist.lan: memberlist: Push/Pull with kafka-eap-ip-10-27-52-135 failed: dial tcp 10.27.52.135:8301: i/o timeout 2021-04-17T01:32:33.412Z [ERROR] agent.server.memberlist.lan: memberlist: Push/Pull with kafka-eap-ip-10-27-53-177 failed: dial tcp 10.27.53.177:8301: i/o timeout