2021-04-16T16:02:44.400Z [WARN] agent: error getting server health from server: server=consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal error="context deadline exceeded" 2021-04-16T16:02:44.892Z [WARN] agent.server.raft: failed to contact: server-id=0bd86151-612d-47e6-a5c7-08287c3515e5 time=2.500096153s 2021-04-16T16:02:46.400Z [WARN] agent: error getting server health from server: server=consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal error="context deadline exceeded" 2021-04-16T16:02:47.373Z [WARN] agent.server.raft: failed to contact: server-id=0bd86151-612d-47e6-a5c7-08287c3515e5 time=4.980475514s 2021-04-16T16:02:48.368Z [INFO] agent.server.serf.lan: serf: EventMemberFailed: consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal 10.27.54.207 2021-04-16T16:02:48.368Z [INFO] agent.server: Removing LAN server: server="consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal (Addr: tcp/10.27.54.207:8300) (DC: apiq-dev-us1-wei-eap)" 2021-04-16T16:02:48.368Z [INFO] agent.server: member failed, marking health critical: member=consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal 2021-04-16T16:02:48.400Z [WARN] agent: error getting server health from server: server=consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal error="context deadline exceeded" 2021-04-16T16:02:49.846Z [WARN] agent.server.raft: failed to contact: server-id=0bd86151-612d-47e6-a5c7-08287c3515e5 time=7.453951964s 2021-04-16T16:02:50.400Z [WARN] agent: error getting server health from server: server=consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal error="context deadline exceeded" 2021-04-16T16:02:52.400Z [WARN] agent: error getting server health from server: server=consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal error="context deadline exceeded" 2021-04-16T16:02:52.442Z [INFO] agent.server.raft: aborting pipeline replication: peer="{Voter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.207:8300}" 2021-04-16T16:02:52.518Z [WARN] agent.server.raft: unable to get address for server, using fallback address: id=0bd86151-612d-47e6-a5c7-08287c3515e5 fallback=10.27.54.207:8300 error="Could not find address for server id 0bd86151-612d-47e6-a5c7-08287c3515e5" 2021-04-16T16:02:52.520Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.27.54.207:8300 error="read tcp 10.27.52.42:37521->10.27.54.207:8300: i/o timeout" 2021-04-16T16:02:52.530Z [WARN] agent.server.raft: unable to get address for server, using fallback address: id=0bd86151-612d-47e6-a5c7-08287c3515e5 fallback=10.27.54.207:8300 error="Could not find address for server id 0bd86151-612d-47e6-a5c7-08287c3515e5" 2021-04-16T16:02:54.400Z [WARN] agent: error getting server health from server: server=consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal error="context deadline exceeded" 2021-04-16T16:02:54.942Z [WARN] agent.server.raft: failed to contact: server-id=0bd86151-612d-47e6-a5c7-08287c3515e5 time=2.500095569s 2021-04-16T16:02:56.400Z [WARN] agent: error getting server health from server: server=consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal error="context deadline exceeded" 2021-04-16T16:02:57.385Z [WARN] agent.server.raft: failed to contact: server-id=0bd86151-612d-47e6-a5c7-08287c3515e5 time=4.943136554s 2021-04-16T16:02:57.400Z [INFO] agent.server.autopilot: Attempting removal of failed server node: id=0bd86151-612d-47e6-a5c7-08287c3515e5 name=consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal address=10.27.54.207:8300 2021-04-16T16:02:57.400Z [INFO] agent.server.serf.lan: serf: EventMemberLeave (forced): consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal 10.27.54.207 2021-04-16T16:02:57.400Z [INFO] agent.server: Removing LAN server: server="consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal (Addr: tcp/10.27.54.207:8300) (DC: apiq-dev-us1-wei-eap)" 2021-04-16T16:02:57.400Z [INFO] agent.server.raft: updating configuration: command=RemoveServer server-id=0bd86151-612d-47e6-a5c7-08287c3515e5 server-addr= servers="[{Suffrage:Voter ID:659cafd8-5944-ecc0-39f4-9639d6f91395 Address:10.27.52.42:8300} {Suffrage:Voter ID:607ffdfd-64be-90b8-2ee1-f65c801aef7a Address:10.27.53.99:8300}]" 2021-04-16T16:02:57.402Z [INFO] agent.server.raft: removed peer, stopping replication: peer=0bd86151-612d-47e6-a5c7-08287c3515e5 last-index=244887 2021-04-16T16:02:57.406Z [INFO] agent.server.autopilot: removed server: id=0bd86151-612d-47e6-a5c7-08287c3515e5 2021-04-16T16:02:57.406Z [INFO] agent.server: deregistering member: member=consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal reason=left 2021-04-16T16:02:58.400Z [WARN] agent: error getting server health from server: server=consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal error="context deadline exceeded" 2021-04-16T16:03:02.518Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Voter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.207:8300}" error="dial tcp 10.27.52.42:0->10.27.54.207:8300: i/o timeout" 2021-04-16T16:03:02.528Z [WARN] agent.server.raft: unable to get address for server, using fallback address: id=0bd86151-612d-47e6-a5c7-08287c3515e5 fallback=10.27.54.207:8300 error="Could not find address for server id 0bd86151-612d-47e6-a5c7-08287c3515e5" 2021-04-16T16:03:02.531Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.27.54.207:8300 error="dial tcp 10.27.52.42:0->10.27.54.207:8300: i/o timeout" 2021-04-16T16:03:03.027Z [ERROR] agent: yamux: keepalive failed: i/o deadline reached 2021-04-16T16:03:03.027Z [WARN] agent: error getting server health from server: server=consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal error="rpc error making call: EOF" 2021-04-16T16:03:03.027Z [WARN] agent: error getting server health from server: server=consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal error="rpc error making call: EOF" 2021-04-16T16:03:03.027Z [WARN] agent: error getting server health from server: server=consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal error="rpc error making call: EOF" 2021-04-16T16:03:03.027Z [WARN] agent: error getting server health from server: server=consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal error="rpc error making call: EOF" 2021-04-16T16:03:03.027Z [WARN] agent: error getting server health from server: server=consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal error="rpc error making call: EOF" 2021-04-16T16:03:03.027Z [WARN] agent: error getting server health from server: server=consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal error="rpc error making call: EOF" 2021-04-16T16:03:03.027Z [WARN] agent: error getting server health from server: server=consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal error="rpc error making call: EOF" 2021-04-16T16:03:03.027Z [WARN] agent: error getting server health from server: server=consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal error="rpc error making call: EOF" 2021-04-16T16:03:03.302Z [ERROR] agent.server: yamux: keepalive failed: i/o deadline reached 2021-04-16T16:03:03.302Z [ERROR] agent.server.rpc: multiplex conn accept failed: conn=from=10.27.54.207:43625 error="keepalive timeout" 2021-04-16T16:03:03.446Z [WARN] agent.server.raft: unable to get address for server, using fallback address: id=0bd86151-612d-47e6-a5c7-08287c3515e5 fallback=10.27.54.207:8300 error="Could not find address for server id 0bd86151-612d-47e6-a5c7-08287c3515e5" 2021-04-16T16:03:12.528Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Voter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.207:8300}" error="dial tcp 10.27.52.42:0->10.27.54.207:8300: i/o timeout" 2021-04-16T16:03:13.447Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.27.54.207:8300 error="dial tcp 10.27.52.42:0->10.27.54.207:8300: i/o timeout" 2021-04-16T16:03:20.163Z [INFO] agent.server.serf.wan: serf: EventMemberLeave: consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal.apiq-dev-us1-wei-eap 10.27.54.207 2021-04-16T16:03:20.163Z [INFO] agent.server: Handled event for server in area: event=member-leave server=consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal.apiq-dev-us1-wei-eap area=wan 2021-04-16T16:06:51.222Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-eap-ip-10-27-54-17-ip-10-27-54-17.us-west-2.compute.internal 10.27.54.17 2021-04-16T16:06:51.222Z [INFO] agent.server: Adding LAN server: server="consul-eap-ip-10-27-54-17-ip-10-27-54-17.us-west-2.compute.internal (Addr: tcp/10.27.54.17:8300) (DC: apiq-dev-us1-wei-eap)" 2021-04-16T16:06:51.222Z [INFO] agent.server.raft: updating configuration: command=AddNonvoter server-id=0bd86151-612d-47e6-a5c7-08287c3515e5 server-addr=10.27.54.17:8300 servers="[{Suffrage:Voter ID:659cafd8-5944-ecc0-39f4-9639d6f91395 Address:10.27.52.42:8300} {Suffrage:Voter ID:607ffdfd-64be-90b8-2ee1-f65c801aef7a Address:10.27.53.99:8300} {Suffrage:Nonvoter ID:0bd86151-612d-47e6-a5c7-08287c3515e5 Address:10.27.54.17:8300}]" 2021-04-16T16:06:51.225Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul-eap-ip-10-27-54-17-ip-10-27-54-17.us-west-2.compute.internal.apiq-dev-us1-wei-eap 10.27.54.17 2021-04-16T16:06:51.225Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul-eap-ip-10-27-54-17-ip-10-27-54-17.us-west-2.compute.internal.apiq-dev-us1-wei-eap area=wan 2021-04-16T16:06:51.227Z [INFO] agent.server.raft: added peer, starting replication: peer=0bd86151-612d-47e6-a5c7-08287c3515e5 2021-04-16T16:06:51.230Z [INFO] agent.server: member joined, marking health alive: member=consul-eap-ip-10-27-54-17-ip-10-27-54-17.us-west-2.compute.internal 2021-04-16T16:06:51.231Z [INFO] agent.server.autopilot: Attempting removal of failed server node: id=0bd86151-612d-47e6-a5c7-08287c3515e5 name=consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal address=10.27.54.207:8300 2021-04-16T16:06:51.233Z [WARN] agent.server.raft: appendEntries rejected, sending older logs: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.17:8300}" next=244882 2021-04-16T16:06:52.925Z [INFO] agent.server.raft: pipelining replication: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.17:8300}" 2021-04-16T16:06:57.400Z [INFO] agent.server.autopilot: Attempting removal of failed server node: id=0bd86151-612d-47e6-a5c7-08287c3515e5 name=consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal address=10.27.54.207:8300 2021-04-16T16:07:07.400Z [INFO] agent.server.autopilot: Promoting server: id=0bd86151-612d-47e6-a5c7-08287c3515e5 address=10.27.54.17:8300 name=consul-eap-ip-10-27-54-17-ip-10-27-54-17.us-west-2.compute.internal 2021-04-16T16:07:07.400Z [INFO] agent.server.raft: updating configuration: command=AddStaging server-id=0bd86151-612d-47e6-a5c7-08287c3515e5 server-addr=10.27.54.17:8300 servers="[{Suffrage:Voter ID:659cafd8-5944-ecc0-39f4-9639d6f91395 Address:10.27.52.42:8300} {Suffrage:Voter ID:607ffdfd-64be-90b8-2ee1-f65c801aef7a Address:10.27.53.99:8300} {Suffrage:Voter ID:0bd86151-612d-47e6-a5c7-08287c3515e5 Address:10.27.54.17:8300}]" 2021-04-16T16:07:08.820Z [INFO] agent.server.raft: updating configuration: command=RemoveServer server-id=0bd86151-612d-47e6-a5c7-08287c3515e5 server-addr= servers="[{Suffrage:Voter ID:659cafd8-5944-ecc0-39f4-9639d6f91395 Address:10.27.52.42:8300} {Suffrage:Voter ID:607ffdfd-64be-90b8-2ee1-f65c801aef7a Address:10.27.53.99:8300}]" 2021-04-16T16:07:08.822Z [INFO] agent.server.raft: removed peer, stopping replication: peer=0bd86151-612d-47e6-a5c7-08287c3515e5 last-index=244953 2021-04-16T16:07:08.822Z [INFO] agent.server.raft: aborting pipeline replication: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.17:8300}" 2021-04-16T16:07:08.824Z [INFO] agent.server.autopilot: removed server: id=0bd86151-612d-47e6-a5c7-08287c3515e5 2021-04-16T16:07:21.679Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.17:8300 leader=10.27.52.42:8300 2021-04-16T16:07:29.069Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.17:8300 leader=10.27.52.42:8300 2021-04-16T16:07:38.173Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.17:8300 leader=10.27.52.42:8300 2021-04-16T16:07:43.689Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.17:8300 leader=10.27.52.42:8300 2021-04-16T16:07:49.969Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.17:8300 leader=10.27.52.42:8300 2021-04-16T16:07:58.654Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.17:8300 leader=10.27.52.42:8300 2021-04-16T16:08:04.254Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.17:8300 leader=10.27.52.42:8300 2021-04-16T16:08:08.821Z [INFO] agent.server.raft: updating configuration: command=AddNonvoter server-id=0bd86151-612d-47e6-a5c7-08287c3515e5 server-addr=10.27.54.17:8300 servers="[{Suffrage:Voter ID:659cafd8-5944-ecc0-39f4-9639d6f91395 Address:10.27.52.42:8300} {Suffrage:Voter ID:607ffdfd-64be-90b8-2ee1-f65c801aef7a Address:10.27.53.99:8300} {Suffrage:Nonvoter ID:0bd86151-612d-47e6-a5c7-08287c3515e5 Address:10.27.54.17:8300}]" 2021-04-16T16:08:08.822Z [INFO] agent.server.raft: added peer, starting replication: peer=0bd86151-612d-47e6-a5c7-08287c3515e5 2021-04-16T16:08:08.823Z [ERROR] agent.server.raft: peer has newer term, stopping replication: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.17:8300}" 2021-04-16T16:08:08.823Z [INFO] agent.server.raft: entering follower state: follower="Node at 10.27.52.42:8300 [Follower]" leader= 2021-04-16T16:08:08.823Z [INFO] agent.server.raft: aborting pipeline replication: peer="{Voter 607ffdfd-64be-90b8-2ee1-f65c801aef7a 10.27.53.99:8300}" 2021-04-16T16:08:08.823Z [ERROR] agent.server.autopilot: failed to add raft non-voting peer: id=0bd86151-612d-47e6-a5c7-08287c3515e5 address=10.27.54.17:8300 error="leadership lost while committing log" 2021-04-16T16:08:08.823Z [ERROR] agent.server: failed to reconcile member: member="{consul-eap-ip-10-27-54-17-ip-10-27-54-17.us-west-2.compute.internal 10.27.54.17 8301 map[acls:0 build:1.9.3:f55da930 dc:apiq-dev-us1-wei-eap expect:3 ft_fs:1 ft_si:1 id:0bd86151-612d-47e6-a5c7-08287c3515e5 port:8300 raft_vsn:3 role:consul segment: use_tls:1 vsn:2 vsn_max:3 vsn_min:2 wan_join_port:8302] alive 1 5 2 2 5 4}" error="leadership lost while committing log" 2021-04-16T16:08:08.823Z [ERROR] agent.server.autopilot: failed to remove raft server: id=0bd86151-612d-47e6-a5c7-08287c3515e5 error="node is not the leader" 2021-04-16T16:08:08.823Z [ERROR] agent.server: failed to reconcile member: member="{consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal 10.27.54.207 8301 map[acls:0 build:1.9.3:f55da930 dc:apiq-dev-us1-wei-eap expect:3 ft_fs:1 ft_si:1 id:0bd86151-612d-47e6-a5c7-08287c3515e5 port:8300 raft_vsn:3 role:consul segment: use_tls:1 vsn:2 vsn_max:3 vsn_min:2 wan_join_port:8302] left 1 5 2 2 5 4}" error="node is not the leader" 2021-04-16T16:08:08.824Z [INFO] agent.server: cluster leadership lost 2021-04-16T16:08:08.999Z [WARN] agent.server.coordinate: Batch update failed: error="node is not the leader" 2021-04-16T16:08:13.374Z [WARN] agent.server.raft: rejecting vote request since our last index is greater: candidate=10.27.54.17:8300 last-index=244967 last-candidate-index=244953 2021-04-16T16:08:16.140Z [WARN] agent.server.raft: heartbeat timeout reached, starting election: last-leader= 2021-04-16T16:08:16.140Z [INFO] agent.server.raft: entering candidate state: node="Node at 10.27.52.42:8300 [Candidate]" term=11 2021-04-16T16:08:16.199Z [INFO] agent.server.raft: election won: tally=2 2021-04-16T16:08:16.199Z [INFO] agent.server.raft: entering leader state: leader="Node at 10.27.52.42:8300 [Leader]" 2021-04-16T16:08:16.199Z [INFO] agent.server.raft: added peer, starting replication: peer=607ffdfd-64be-90b8-2ee1-f65c801aef7a 2021-04-16T16:08:16.199Z [INFO] agent.server.raft: added peer, starting replication: peer=0bd86151-612d-47e6-a5c7-08287c3515e5 2021-04-16T16:08:16.199Z [INFO] agent.server: cluster leadership acquired 2021-04-16T16:08:16.199Z [INFO] agent.server: New leader elected: payload=consul-eap-ip-10-27-52-42-ip-10-27-52-42.us-west-2.compute.internal 2021-04-16T16:08:16.200Z [INFO] agent.server.raft: pipelining replication: peer="{Voter 607ffdfd-64be-90b8-2ee1-f65c801aef7a 10.27.53.99:8300}" 2021-04-16T16:08:16.201Z [WARN] agent.server.raft: appendEntries rejected, sending older logs: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.17:8300}" next=244954 2021-04-16T16:08:16.203Z [INFO] agent.server.raft: pipelining replication: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.17:8300}" 2021-04-16T16:08:16.208Z [INFO] agent.leader: started routine: routine="federation state anti-entropy" 2021-04-16T16:08:16.208Z [INFO] agent.leader: started routine: routine="federation state pruning" 2021-04-16T16:08:16.209Z [INFO] agent.leader: started routine: routine="intermediate cert renew watch" 2021-04-16T16:08:16.209Z [INFO] agent.leader: started routine: routine="CA root pruning" 2021-04-16T16:08:16.209Z [INFO] agent.server.raft: updating configuration: command=RemoveServer server-id=0bd86151-612d-47e6-a5c7-08287c3515e5 server-addr= servers="[{Suffrage:Voter ID:659cafd8-5944-ecc0-39f4-9639d6f91395 Address:10.27.52.42:8300} {Suffrage:Voter ID:607ffdfd-64be-90b8-2ee1-f65c801aef7a Address:10.27.53.99:8300}]" 2021-04-16T16:08:16.211Z [INFO] agent.server.raft: removed peer, stopping replication: peer=0bd86151-612d-47e6-a5c7-08287c3515e5 last-index=244970 2021-04-16T16:08:16.211Z [INFO] agent.server.raft: aborting pipeline replication: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.17:8300}" 2021-04-16T16:08:16.216Z [INFO] agent.server.autopilot: removed server: id=0bd86151-612d-47e6-a5c7-08287c3515e5 2021-04-16T16:08:25.342Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.17:8300 leader=10.27.52.42:8300 2021-04-16T16:08:33.543Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.17:8300 leader=10.27.52.42:8300 2021-04-16T16:08:41.572Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.17:8300 leader=10.27.52.42:8300 2021-04-16T16:08:50.671Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.17:8300 leader=10.27.52.42:8300 2021-04-16T16:08:57.948Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.17:8300 leader=10.27.52.42:8300 2021-04-16T16:09:04.653Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.17:8300 leader=10.27.52.42:8300 2021-04-16T16:09:10.934Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.17:8300 leader=10.27.52.42:8300 2021-04-16T16:09:16.203Z [INFO] agent.server.raft: updating configuration: command=AddNonvoter server-id=0bd86151-612d-47e6-a5c7-08287c3515e5 server-addr=10.27.54.17:8300 servers="[{Suffrage:Voter ID:659cafd8-5944-ecc0-39f4-9639d6f91395 Address:10.27.52.42:8300} {Suffrage:Voter ID:607ffdfd-64be-90b8-2ee1-f65c801aef7a Address:10.27.53.99:8300} {Suffrage:Nonvoter ID:0bd86151-612d-47e6-a5c7-08287c3515e5 Address:10.27.54.17:8300}]" 2021-04-16T16:09:16.205Z [INFO] agent.server.raft: added peer, starting replication: peer=0bd86151-612d-47e6-a5c7-08287c3515e5 2021-04-16T16:09:16.205Z [ERROR] agent.server.raft: peer has newer term, stopping replication: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.17:8300}" 2021-04-16T16:09:16.205Z [INFO] agent.server.raft: entering follower state: follower="Node at 10.27.52.42:8300 [Follower]" leader= 2021-04-16T16:09:16.205Z [INFO] agent.server.raft: aborting pipeline replication: peer="{Voter 607ffdfd-64be-90b8-2ee1-f65c801aef7a 10.27.53.99:8300}" 2021-04-16T16:09:16.205Z [ERROR] agent.server.autopilot: failed to add raft non-voting peer: id=0bd86151-612d-47e6-a5c7-08287c3515e5 address=10.27.54.17:8300 error="leadership lost while committing log" 2021-04-16T16:09:16.205Z [ERROR] agent.server: failed to reconcile member: member="{consul-eap-ip-10-27-54-17-ip-10-27-54-17.us-west-2.compute.internal 10.27.54.17 8301 map[acls:0 build:1.9.3:f55da930 dc:apiq-dev-us1-wei-eap expect:3 ft_fs:1 ft_si:1 id:0bd86151-612d-47e6-a5c7-08287c3515e5 port:8300 raft_vsn:3 role:consul segment: use_tls:1 vsn:2 vsn_max:3 vsn_min:2 wan_join_port:8302] alive 1 5 2 2 5 4}" error="leadership lost while committing log" 2021-04-16T16:09:16.205Z [ERROR] agent.server.autopilot: failed to remove raft server: id=0bd86151-612d-47e6-a5c7-08287c3515e5 error="node is not the leader" 2021-04-16T16:09:16.206Z [ERROR] agent.server: failed to reconcile member: member="{consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal 10.27.54.207 8301 map[acls:0 build:1.9.3:f55da930 dc:apiq-dev-us1-wei-eap expect:3 ft_fs:1 ft_si:1 id:0bd86151-612d-47e6-a5c7-08287c3515e5 port:8300 raft_vsn:3 role:consul segment: use_tls:1 vsn:2 vsn_max:3 vsn_min:2 wan_join_port:8302] left 1 5 2 2 5 4}" error="node is not the leader" 2021-04-16T16:09:16.206Z [INFO] agent.server: cluster leadership lost 2021-04-16T16:09:16.841Z [WARN] agent.server.raft: rejecting vote request since our last index is greater: candidate=10.27.54.17:8300 last-index=244993 last-candidate-index=244970 2021-04-16T16:09:23.414Z [WARN] agent.server.raft: rejecting vote request since our last index is greater: candidate=10.27.54.17:8300 last-index=244993 last-candidate-index=244970 2021-04-16T16:09:25.835Z [WARN] agent.server.raft: heartbeat timeout reached, starting election: last-leader= 2021-04-16T16:09:25.835Z [INFO] agent.server.raft: entering candidate state: node="Node at 10.27.52.42:8300 [Candidate]" term=21 2021-04-16T16:09:25.845Z [INFO] agent.server.raft: election won: tally=2 2021-04-16T16:09:25.845Z [INFO] agent.server.raft: entering leader state: leader="Node at 10.27.52.42:8300 [Leader]" 2021-04-16T16:09:25.845Z [INFO] agent.server.raft: added peer, starting replication: peer=607ffdfd-64be-90b8-2ee1-f65c801aef7a 2021-04-16T16:09:25.845Z [INFO] agent.server: cluster leadership acquired 2021-04-16T16:09:25.845Z [INFO] agent.server.raft: added peer, starting replication: peer=0bd86151-612d-47e6-a5c7-08287c3515e5 2021-04-16T16:09:25.845Z [INFO] agent.server: New leader elected: payload=consul-eap-ip-10-27-52-42-ip-10-27-52-42.us-west-2.compute.internal 2021-04-16T16:09:25.846Z [INFO] agent.server.raft: pipelining replication: peer="{Voter 607ffdfd-64be-90b8-2ee1-f65c801aef7a 10.27.53.99:8300}" 2021-04-16T16:09:25.847Z [WARN] agent.server.raft: appendEntries rejected, sending older logs: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.17:8300}" next=244971 2021-04-16T16:09:25.849Z [INFO] agent.server.raft: pipelining replication: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.17:8300}" 2021-04-16T16:09:25.852Z [INFO] agent.leader: started routine: routine="federation state anti-entropy" 2021-04-16T16:09:25.852Z [INFO] agent.leader: started routine: routine="federation state pruning" 2021-04-16T16:09:25.853Z [INFO] agent.leader: started routine: routine="intermediate cert renew watch" 2021-04-16T16:09:25.853Z [INFO] agent.leader: started routine: routine="CA root pruning" 2021-04-16T16:09:25.853Z [INFO] agent.server.raft: updating configuration: command=RemoveServer server-id=0bd86151-612d-47e6-a5c7-08287c3515e5 server-addr= servers="[{Suffrage:Voter ID:659cafd8-5944-ecc0-39f4-9639d6f91395 Address:10.27.52.42:8300} {Suffrage:Voter ID:607ffdfd-64be-90b8-2ee1-f65c801aef7a Address:10.27.53.99:8300}]" 2021-04-16T16:09:25.854Z [INFO] agent.server.raft: removed peer, stopping replication: peer=0bd86151-612d-47e6-a5c7-08287c3515e5 last-index=244996 2021-04-16T16:09:25.855Z [INFO] agent.server.raft: aborting pipeline replication: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.17:8300}" 2021-04-16T16:09:25.858Z [INFO] agent.server.autopilot: removed server: id=0bd86151-612d-47e6-a5c7-08287c3515e5 2021-04-16T16:09:31.469Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.17:8300 leader=10.27.52.42:8300 2021-04-16T16:09:40.250Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.17:8300 leader=10.27.52.42:8300 2021-04-16T16:09:45.307Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.17:8300 leader=10.27.52.42:8300 2021-04-16T16:09:52.500Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.17:8300 leader=10.27.52.42:8300 2021-04-16T16:10:00.553Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.17:8300 leader=10.27.52.42:8300 2021-04-16T16:10:08.152Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.17:8300 leader=10.27.52.42:8300 2021-04-16T16:10:15.341Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.17:8300 leader=10.27.52.42:8300 2021-04-16T16:10:19.813Z [ERROR] agent.server: error performing anti-entropy sync of federation state: error="context canceled" 2021-04-16T16:10:23.917Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.17:8300 leader=10.27.52.42:8300 2021-04-16T16:10:25.849Z [INFO] agent.server.raft: updating configuration: command=AddNonvoter server-id=0bd86151-612d-47e6-a5c7-08287c3515e5 server-addr=10.27.54.17:8300 servers="[{Suffrage:Voter ID:659cafd8-5944-ecc0-39f4-9639d6f91395 Address:10.27.52.42:8300} {Suffrage:Voter ID:607ffdfd-64be-90b8-2ee1-f65c801aef7a Address:10.27.53.99:8300} {Suffrage:Nonvoter ID:0bd86151-612d-47e6-a5c7-08287c3515e5 Address:10.27.54.17:8300}]" 2021-04-16T16:10:25.850Z [INFO] agent.server.raft: added peer, starting replication: peer=0bd86151-612d-47e6-a5c7-08287c3515e5 2021-04-16T16:10:25.851Z [ERROR] agent.server.raft: peer has newer term, stopping replication: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.17:8300}" 2021-04-16T16:10:25.851Z [INFO] agent.server.raft: entering follower state: follower="Node at 10.27.52.42:8300 [Follower]" leader= 2021-04-16T16:10:25.851Z [INFO] agent.server.raft: aborting pipeline replication: peer="{Voter 607ffdfd-64be-90b8-2ee1-f65c801aef7a 10.27.53.99:8300}" 2021-04-16T16:10:25.851Z [ERROR] agent.server.autopilot: failed to add raft non-voting peer: id=0bd86151-612d-47e6-a5c7-08287c3515e5 address=10.27.54.17:8300 error="leadership lost while committing log" 2021-04-16T16:10:25.851Z [ERROR] agent.server: failed to reconcile member: member="{consul-eap-ip-10-27-54-17-ip-10-27-54-17.us-west-2.compute.internal 10.27.54.17 8301 map[acls:0 build:1.9.3:f55da930 dc:apiq-dev-us1-wei-eap expect:3 ft_fs:1 ft_si:1 id:0bd86151-612d-47e6-a5c7-08287c3515e5 port:8300 raft_vsn:3 role:consul segment: use_tls:1 vsn:2 vsn_max:3 vsn_min:2 wan_join_port:8302] alive 1 5 2 2 5 4}" error="leadership lost while committing log" 2021-04-16T16:10:25.851Z [ERROR] agent.server.autopilot: failed to remove raft server: id=0bd86151-612d-47e6-a5c7-08287c3515e5 error="node is not the leader" 2021-04-16T16:10:25.851Z [ERROR] agent.server: failed to reconcile member: member="{consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal 10.27.54.207 8301 map[acls:0 build:1.9.3:f55da930 dc:apiq-dev-us1-wei-eap expect:3 ft_fs:1 ft_si:1 id:0bd86151-612d-47e6-a5c7-08287c3515e5 port:8300 raft_vsn:3 role:consul segment: use_tls:1 vsn:2 vsn_max:3 vsn_min:2 wan_join_port:8302] left 1 5 2 2 5 4}" error="node is not the leader" 2021-04-16T16:10:25.852Z [INFO] agent.server: cluster leadership lost 2021-04-16T16:10:29.112Z [WARN] agent.server.coordinate: Batch update failed: error="node is not the leader" 2021-04-16T16:10:30.440Z [WARN] agent.server.raft: rejecting vote request since our last index is greater: candidate=10.27.54.17:8300 last-index=245013 last-candidate-index=244996 2021-04-16T16:10:31.184Z [WARN] agent.server.raft: heartbeat timeout reached, starting election: last-leader= 2021-04-16T16:10:31.184Z [INFO] agent.server.raft: entering candidate state: node="Node at 10.27.52.42:8300 [Candidate]" term=31 2021-04-16T16:10:31.195Z [INFO] agent.server.raft: election won: tally=2 2021-04-16T16:10:31.195Z [INFO] agent.server.raft: entering leader state: leader="Node at 10.27.52.42:8300 [Leader]" 2021-04-16T16:10:31.195Z [INFO] agent.server.raft: added peer, starting replication: peer=607ffdfd-64be-90b8-2ee1-f65c801aef7a 2021-04-16T16:10:31.195Z [INFO] agent.server.raft: added peer, starting replication: peer=0bd86151-612d-47e6-a5c7-08287c3515e5 2021-04-16T16:10:31.195Z [INFO] agent.server: cluster leadership acquired 2021-04-16T16:10:31.195Z [INFO] agent.server: New leader elected: payload=consul-eap-ip-10-27-52-42-ip-10-27-52-42.us-west-2.compute.internal 2021-04-16T16:10:31.196Z [INFO] agent.server.raft: pipelining replication: peer="{Voter 607ffdfd-64be-90b8-2ee1-f65c801aef7a 10.27.53.99:8300}" 2021-04-16T16:10:31.197Z [WARN] agent.server.raft: appendEntries rejected, sending older logs: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.17:8300}" next=244997 2021-04-16T16:10:31.200Z [INFO] agent.server.raft: pipelining replication: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.17:8300}" 2021-04-16T16:10:31.201Z [INFO] agent.leader: started routine: routine="federation state anti-entropy" 2021-04-16T16:10:31.201Z [INFO] agent.leader: started routine: routine="federation state pruning" 2021-04-16T16:10:31.202Z [INFO] agent.leader: started routine: routine="intermediate cert renew watch" 2021-04-16T16:10:31.202Z [INFO] agent.leader: started routine: routine="CA root pruning" 2021-04-16T16:10:31.202Z [INFO] agent.server.raft: updating configuration: command=RemoveServer server-id=0bd86151-612d-47e6-a5c7-08287c3515e5 server-addr= servers="[{Suffrage:Voter ID:659cafd8-5944-ecc0-39f4-9639d6f91395 Address:10.27.52.42:8300} {Suffrage:Voter ID:607ffdfd-64be-90b8-2ee1-f65c801aef7a Address:10.27.53.99:8300}]" 2021-04-16T16:10:31.204Z [INFO] agent.server.raft: removed peer, stopping replication: peer=0bd86151-612d-47e6-a5c7-08287c3515e5 last-index=245016 2021-04-16T16:10:31.204Z [INFO] agent.server.raft: aborting pipeline replication: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.17:8300}" 2021-04-16T16:10:31.207Z [INFO] agent.server.autopilot: removed server: id=0bd86151-612d-47e6-a5c7-08287c3515e5 2021-04-16T16:10:37.441Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.17:8300 leader=10.27.52.42:8300 2021-04-16T16:10:45.663Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.17:8300 leader=10.27.52.42:8300 2021-04-16T16:10:55.360Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.17:8300 leader=10.27.52.42:8300 2021-04-16T16:11:04.332Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.17:8300 leader=10.27.52.42:8300 2021-04-16T16:11:10.733Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.17:8300 leader=10.27.52.42:8300 2021-04-16T16:11:18.137Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.17:8300 leader=10.27.52.42:8300 2021-04-16T16:11:24.374Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.17:8300 leader=10.27.52.42:8300 2021-04-16T16:11:31.200Z [INFO] agent.server.raft: updating configuration: command=AddNonvoter server-id=0bd86151-612d-47e6-a5c7-08287c3515e5 server-addr=10.27.54.17:8300 servers="[{Suffrage:Voter ID:659cafd8-5944-ecc0-39f4-9639d6f91395 Address:10.27.52.42:8300} {Suffrage:Voter ID:607ffdfd-64be-90b8-2ee1-f65c801aef7a Address:10.27.53.99:8300} {Suffrage:Nonvoter ID:0bd86151-612d-47e6-a5c7-08287c3515e5 Address:10.27.54.17:8300}]" 2021-04-16T16:11:31.202Z [INFO] agent.server.raft: added peer, starting replication: peer=0bd86151-612d-47e6-a5c7-08287c3515e5 2021-04-16T16:11:31.202Z [ERROR] agent.server.raft: peer has newer term, stopping replication: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.17:8300}" 2021-04-16T16:11:31.203Z [INFO] agent.server.raft: entering follower state: follower="Node at 10.27.52.42:8300 [Follower]" leader= 2021-04-16T16:11:31.203Z [INFO] agent.server.raft: aborting pipeline replication: peer="{Voter 607ffdfd-64be-90b8-2ee1-f65c801aef7a 10.27.53.99:8300}" 2021-04-16T16:11:31.203Z [ERROR] agent.server.autopilot: failed to add raft non-voting peer: id=0bd86151-612d-47e6-a5c7-08287c3515e5 address=10.27.54.17:8300 error="leadership lost while committing log" 2021-04-16T16:11:31.203Z [ERROR] agent.server: failed to reconcile member: member="{consul-eap-ip-10-27-54-17-ip-10-27-54-17.us-west-2.compute.internal 10.27.54.17 8301 map[acls:0 build:1.9.3:f55da930 dc:apiq-dev-us1-wei-eap expect:3 ft_fs:1 ft_si:1 id:0bd86151-612d-47e6-a5c7-08287c3515e5 port:8300 raft_vsn:3 role:consul segment: use_tls:1 vsn:2 vsn_max:3 vsn_min:2 wan_join_port:8302] alive 1 5 2 2 5 4}" error="leadership lost while committing log" 2021-04-16T16:11:31.203Z [INFO] agent.server: cluster leadership lost 2021-04-16T16:11:32.092Z [WARN] agent.server.raft: rejecting vote request since our last index is greater: candidate=10.27.54.17:8300 last-index=245031 last-candidate-index=245016 2021-04-16T16:11:37.296Z [WARN] agent.server.raft: heartbeat timeout reached, starting election: last-leader= 2021-04-16T16:11:37.296Z [INFO] agent.server.raft: entering candidate state: node="Node at 10.27.52.42:8300 [Candidate]" term=40 2021-04-16T16:11:37.308Z [INFO] agent.server.raft: election won: tally=2 2021-04-16T16:11:37.308Z [INFO] agent.server.raft: entering leader state: leader="Node at 10.27.52.42:8300 [Leader]" 2021-04-16T16:11:37.308Z [INFO] agent.server.raft: added peer, starting replication: peer=607ffdfd-64be-90b8-2ee1-f65c801aef7a 2021-04-16T16:11:37.308Z [INFO] agent.server.raft: added peer, starting replication: peer=0bd86151-612d-47e6-a5c7-08287c3515e5 2021-04-16T16:11:37.308Z [INFO] agent.server: cluster leadership acquired 2021-04-16T16:11:37.308Z [INFO] agent.server: New leader elected: payload=consul-eap-ip-10-27-52-42-ip-10-27-52-42.us-west-2.compute.internal 2021-04-16T16:11:37.309Z [INFO] agent.server.raft: pipelining replication: peer="{Voter 607ffdfd-64be-90b8-2ee1-f65c801aef7a 10.27.53.99:8300}" 2021-04-16T16:11:37.311Z [WARN] agent.server.raft: appendEntries rejected, sending older logs: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.17:8300}" next=245017 2021-04-16T16:11:37.313Z [INFO] agent.server.raft: pipelining replication: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.17:8300}" 2021-04-16T16:11:37.315Z [INFO] agent.leader: started routine: routine="federation state anti-entropy" 2021-04-16T16:11:37.315Z [INFO] agent.leader: started routine: routine="federation state pruning" 2021-04-16T16:11:37.315Z [INFO] agent.leader: started routine: routine="intermediate cert renew watch" 2021-04-16T16:11:37.316Z [INFO] agent.leader: started routine: routine="CA root pruning" 2021-04-16T16:11:37.316Z [INFO] agent.server.raft: updating configuration: command=RemoveServer server-id=0bd86151-612d-47e6-a5c7-08287c3515e5 server-addr= servers="[{Suffrage:Voter ID:659cafd8-5944-ecc0-39f4-9639d6f91395 Address:10.27.52.42:8300} {Suffrage:Voter ID:607ffdfd-64be-90b8-2ee1-f65c801aef7a Address:10.27.53.99:8300}]" 2021-04-16T16:11:37.317Z [INFO] agent.server.raft: removed peer, stopping replication: peer=0bd86151-612d-47e6-a5c7-08287c3515e5 last-index=245034 2021-04-16T16:11:37.317Z [INFO] agent.server.raft: aborting pipeline replication: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.17:8300}" 2021-04-16T16:11:37.320Z [INFO] agent.server.autopilot: removed server: id=0bd86151-612d-47e6-a5c7-08287c3515e5 2021-04-16T16:11:37.320Z [INFO] agent.server.raft: updating configuration: command=AddNonvoter server-id=0bd86151-612d-47e6-a5c7-08287c3515e5 server-addr=10.27.54.17:8300 servers="[{Suffrage:Voter ID:659cafd8-5944-ecc0-39f4-9639d6f91395 Address:10.27.52.42:8300} {Suffrage:Voter ID:607ffdfd-64be-90b8-2ee1-f65c801aef7a Address:10.27.53.99:8300} {Suffrage:Nonvoter ID:0bd86151-612d-47e6-a5c7-08287c3515e5 Address:10.27.54.17:8300}]" 2021-04-16T16:11:37.323Z [INFO] agent.server.raft: added peer, starting replication: peer=0bd86151-612d-47e6-a5c7-08287c3515e5 2021-04-16T16:11:37.329Z [WARN] agent.server.raft: appendEntries rejected, sending older logs: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.17:8300}" next=245035 2021-04-16T16:11:37.331Z [INFO] agent.server.raft: pipelining replication: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.17:8300}" 2021-04-16T16:11:57.316Z [INFO] agent.server.autopilot: Promoting server: id=0bd86151-612d-47e6-a5c7-08287c3515e5 address=10.27.54.17:8300 name=consul-eap-ip-10-27-54-17-ip-10-27-54-17.us-west-2.compute.internal 2021-04-16T16:11:57.316Z [INFO] agent.server.raft: updating configuration: command=AddStaging server-id=0bd86151-612d-47e6-a5c7-08287c3515e5 server-addr=10.27.54.17:8300 servers="[{Suffrage:Voter ID:659cafd8-5944-ecc0-39f4-9639d6f91395 Address:10.27.52.42:8300} {Suffrage:Voter ID:607ffdfd-64be-90b8-2ee1-f65c801aef7a Address:10.27.53.99:8300} {Suffrage:Voter ID:0bd86151-612d-47e6-a5c7-08287c3515e5 Address:10.27.54.17:8300}]" 2021-04-16T16:11:57.321Z [INFO] agent.server.autopilot: Attempting removal of failed server node: id=0bd86151-612d-47e6-a5c7-08287c3515e5 name=consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal address=10.27.54.207:8300 2021-04-16T16:12:07.316Z [INFO] agent.server.autopilot: Attempting removal of failed server node: id=0bd86151-612d-47e6-a5c7-08287c3515e5 name=consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal address=10.27.54.207:8300 2021-04-16T16:12:17.316Z [INFO] agent.server.autopilot: Attempting removal of failed server node: id=0bd86151-612d-47e6-a5c7-08287c3515e5 name=consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal address=10.27.54.207:8300 2021-04-16T16:12:27.316Z [INFO] agent.server.autopilot: Attempting removal of failed server node: id=0bd86151-612d-47e6-a5c7-08287c3515e5 name=consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal address=10.27.54.207:8300 2021-04-16T16:12:37.314Z [INFO] agent.server.raft: updating configuration: command=RemoveServer server-id=0bd86151-612d-47e6-a5c7-08287c3515e5 server-addr= servers="[{Suffrage:Voter ID:659cafd8-5944-ecc0-39f4-9639d6f91395 Address:10.27.52.42:8300} {Suffrage:Voter ID:607ffdfd-64be-90b8-2ee1-f65c801aef7a Address:10.27.53.99:8300}]" 2021-04-16T16:12:37.315Z [INFO] agent.server.raft: removed peer, stopping replication: peer=0bd86151-612d-47e6-a5c7-08287c3515e5 last-index=245050 2021-04-16T16:12:37.315Z [INFO] agent.server.raft: aborting pipeline replication: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.17:8300}" 2021-04-16T16:12:37.318Z [INFO] agent.server.autopilot: removed server: id=0bd86151-612d-47e6-a5c7-08287c3515e5 2021-04-16T16:12:37.318Z [INFO] agent.server.raft: updating configuration: command=AddNonvoter server-id=0bd86151-612d-47e6-a5c7-08287c3515e5 server-addr=10.27.54.17:8300 servers="[{Suffrage:Voter ID:659cafd8-5944-ecc0-39f4-9639d6f91395 Address:10.27.52.42:8300} {Suffrage:Voter ID:607ffdfd-64be-90b8-2ee1-f65c801aef7a Address:10.27.53.99:8300} {Suffrage:Nonvoter ID:0bd86151-612d-47e6-a5c7-08287c3515e5 Address:10.27.54.17:8300}]" 2021-04-16T16:12:37.320Z [INFO] agent.server.raft: added peer, starting replication: peer=0bd86151-612d-47e6-a5c7-08287c3515e5 2021-04-16T16:12:42.316Z [WARN] agent: error getting server health from server: server=consul-eap-ip-10-27-54-17-ip-10-27-54-17.us-west-2.compute.internal error="context deadline exceeded" 2021-04-16T16:12:42.978Z [INFO] agent.server.serf.lan: serf: EventMemberFailed: consul-eap-ip-10-27-54-17-ip-10-27-54-17.us-west-2.compute.internal 10.27.54.17 2021-04-16T16:12:42.978Z [INFO] agent.server: Removing LAN server: server="consul-eap-ip-10-27-54-17-ip-10-27-54-17.us-west-2.compute.internal (Addr: tcp/10.27.54.17:8300) (DC: apiq-dev-us1-wei-eap)" 2021-04-16T16:12:42.978Z [INFO] agent.server: member failed, marking health critical: member=consul-eap-ip-10-27-54-17-ip-10-27-54-17.us-west-2.compute.internal 2021-04-16T16:12:44.316Z [WARN] agent: error getting server health from server: server=consul-eap-ip-10-27-54-17-ip-10-27-54-17.us-west-2.compute.internal error="context deadline exceeded" 2021-04-16T16:12:46.316Z [WARN] agent: error getting server health from server: server=consul-eap-ip-10-27-54-17-ip-10-27-54-17.us-west-2.compute.internal error="context deadline exceeded" 2021-04-16T16:12:46.816Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.27.54.17:8300 error="read tcp 10.27.52.42:45269->10.27.54.17:8300: i/o timeout" 2021-04-16T16:12:47.316Z [INFO] agent.server.autopilot: Attempting removal of failed server node: id=0bd86151-612d-47e6-a5c7-08287c3515e5 name=consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal address=10.27.54.207:8300 2021-04-16T16:12:47.320Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.17:8300}" error="dial tcp 10.27.52.42:0->10.27.54.17:8300: i/o timeout" 2021-04-16T16:12:47.331Z [WARN] agent.server.raft: unable to get address for server, using fallback address: id=0bd86151-612d-47e6-a5c7-08287c3515e5 fallback=10.27.54.17:8300 error="Could not find address for server id 0bd86151-612d-47e6-a5c7-08287c3515e5" 2021-04-16T16:12:48.237Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.27.54.17:8300 error="dial tcp 10.27.52.42:0->10.27.54.17:8300: i/o timeout" 2021-04-16T16:12:48.247Z [WARN] agent.server.raft: unable to get address for server, using fallback address: id=0bd86151-612d-47e6-a5c7-08287c3515e5 fallback=10.27.54.17:8300 error="Could not find address for server id 0bd86151-612d-47e6-a5c7-08287c3515e5" 2021-04-16T16:12:56.316Z [WARN] agent: error getting server health from server: server=consul-eap-ip-10-27-54-17-ip-10-27-54-17.us-west-2.compute.internal error="context deadline exceeded" 2021-04-16T16:12:57.316Z [INFO] agent.server.autopilot: Attempting removal of failed server node: id=0bd86151-612d-47e6-a5c7-08287c3515e5 name=consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal address=10.27.54.207:8300 2021-04-16T16:12:57.331Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.17:8300}" error="dial tcp 10.27.52.42:0->10.27.54.17:8300: i/o timeout" 2021-04-16T16:12:57.341Z [WARN] agent.server.raft: unable to get address for server, using fallback address: id=0bd86151-612d-47e6-a5c7-08287c3515e5 fallback=10.27.54.17:8300 error="Could not find address for server id 0bd86151-612d-47e6-a5c7-08287c3515e5" 2021-04-16T16:12:58.247Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.27.54.17:8300 error="dial tcp 10.27.52.42:0->10.27.54.17:8300: i/o timeout" 2021-04-16T16:12:58.908Z [WARN] agent.server.raft: unable to get address for server, using fallback address: id=0bd86151-612d-47e6-a5c7-08287c3515e5 fallback=10.27.54.17:8300 error="Could not find address for server id 0bd86151-612d-47e6-a5c7-08287c3515e5" 2021-04-16T16:13:00.316Z [WARN] agent: error getting server health from server: server=consul-eap-ip-10-27-54-17-ip-10-27-54-17.us-west-2.compute.internal error="context deadline exceeded" 2021-04-16T16:13:07.316Z [INFO] agent.server.autopilot: Attempting removal of failed server node: id=0bd86151-612d-47e6-a5c7-08287c3515e5 name=consul-eap-ip-10-27-54-17-ip-10-27-54-17.us-west-2.compute.internal address=10.27.54.17:8300 2021-04-16T16:13:07.316Z [INFO] agent.server.serf.lan: serf: EventMemberLeave (forced): consul-eap-ip-10-27-54-17-ip-10-27-54-17.us-west-2.compute.internal 10.27.54.17 2021-04-16T16:13:07.317Z [INFO] agent.server: Removing LAN server: server="consul-eap-ip-10-27-54-17-ip-10-27-54-17.us-west-2.compute.internal (Addr: tcp/10.27.54.17:8300) (DC: apiq-dev-us1-wei-eap)" 2021-04-16T16:13:07.317Z [INFO] agent.server.raft: updating configuration: command=RemoveServer server-id=0bd86151-612d-47e6-a5c7-08287c3515e5 server-addr= servers="[{Suffrage:Voter ID:659cafd8-5944-ecc0-39f4-9639d6f91395 Address:10.27.52.42:8300} {Suffrage:Voter ID:607ffdfd-64be-90b8-2ee1-f65c801aef7a Address:10.27.53.99:8300}]" 2021-04-16T16:13:07.318Z [INFO] agent.server.raft: removed peer, stopping replication: peer=0bd86151-612d-47e6-a5c7-08287c3515e5 last-index=245059 2021-04-16T16:13:07.321Z [INFO] agent.server.autopilot: removed server: id=0bd86151-612d-47e6-a5c7-08287c3515e5 2021-04-16T16:13:07.321Z [INFO] agent.server: deregistering member: member=consul-eap-ip-10-27-54-17-ip-10-27-54-17.us-west-2.compute.internal reason=left 2021-04-16T16:13:07.341Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.17:8300}" error="dial tcp 10.27.52.42:0->10.27.54.17:8300: i/o timeout" 2021-04-16T16:13:07.361Z [WARN] agent.server.raft: unable to get address for server, using fallback address: id=0bd86151-612d-47e6-a5c7-08287c3515e5 fallback=10.27.54.17:8300 error="Could not find address for server id 0bd86151-612d-47e6-a5c7-08287c3515e5" 2021-04-16T16:13:07.410Z [ERROR] agent: yamux: keepalive failed: i/o deadline reached 2021-04-16T16:13:07.410Z [WARN] agent: error getting server health from server: server=consul-eap-ip-10-27-54-17-ip-10-27-54-17.us-west-2.compute.internal error="rpc error making call: EOF" 2021-04-16T16:13:07.410Z [WARN] agent: error getting server health from server: server=consul-eap-ip-10-27-54-17-ip-10-27-54-17.us-west-2.compute.internal error="rpc error making call: EOF" 2021-04-16T16:13:07.410Z [WARN] agent: error getting server health from server: server=consul-eap-ip-10-27-54-17-ip-10-27-54-17.us-west-2.compute.internal error="rpc error making call: EOF" 2021-04-16T16:13:07.410Z [WARN] agent: error getting server health from server: server=consul-eap-ip-10-27-54-17-ip-10-27-54-17.us-west-2.compute.internal error="rpc error making call: EOF" 2021-04-16T16:13:07.410Z [WARN] agent: error getting server health from server: server=consul-eap-ip-10-27-54-17-ip-10-27-54-17.us-west-2.compute.internal error="rpc error making call: EOF" 2021-04-16T16:13:08.909Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.27.54.17:8300 error="dial tcp 10.27.52.42:0->10.27.54.17:8300: i/o timeout" 2021-04-16T16:13:09.438Z [WARN] agent.server.raft: unable to get address for server, using fallback address: id=0bd86151-612d-47e6-a5c7-08287c3515e5 fallback=10.27.54.17:8300 error="Could not find address for server id 0bd86151-612d-47e6-a5c7-08287c3515e5" 2021-04-16T16:13:11.483Z [ERROR] agent.server: yamux: keepalive failed: i/o deadline reached 2021-04-16T16:13:11.483Z [ERROR] agent.server.rpc: multiplex conn accept failed: conn=from=10.27.54.17:34389 error="keepalive timeout" 2021-04-16T16:13:11.840Z [INFO] agent.server.serf.wan: serf: EventMemberLeave: consul-eap-ip-10-27-54-17-ip-10-27-54-17.us-west-2.compute.internal.apiq-dev-us1-wei-eap 10.27.54.17 2021-04-16T16:13:11.840Z [INFO] agent.server: Handled event for server in area: event=member-leave server=consul-eap-ip-10-27-54-17-ip-10-27-54-17.us-west-2.compute.internal.apiq-dev-us1-wei-eap area=wan 2021-04-16T16:13:17.361Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.17:8300}" error="dial tcp 10.27.52.42:0->10.27.54.17:8300: i/o timeout" 2021-04-16T16:13:17.401Z [WARN] agent.server.raft: unable to get address for server, using fallback address: id=0bd86151-612d-47e6-a5c7-08287c3515e5 fallback=10.27.54.17:8300 error="Could not find address for server id 0bd86151-612d-47e6-a5c7-08287c3515e5" 2021-04-16T16:13:19.438Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.27.54.17:8300 error="dial tcp 10.27.52.42:0->10.27.54.17:8300: i/o timeout" 2021-04-16T16:13:20.464Z [WARN] agent.server.raft: unable to get address for server, using fallback address: id=0bd86151-612d-47e6-a5c7-08287c3515e5 fallback=10.27.54.17:8300 error="Could not find address for server id 0bd86151-612d-47e6-a5c7-08287c3515e5" 2021-04-16T16:13:23.189Z [ERROR] agent.server: error performing anti-entropy sync of federation state: error="context canceled" 2021-04-16T16:13:27.402Z [ERROR] agent.server.raft: failed to appendEntries to: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.17:8300}" error="dial tcp 10.27.52.42:0->10.27.54.17:8300: i/o timeout" 2021-04-16T16:13:30.464Z [ERROR] agent.server.raft: failed to heartbeat to: peer=10.27.54.17:8300 error="dial tcp 10.27.52.42:0->10.27.54.17:8300: i/o timeout" 2021-04-16T16:14:35.235Z [ERROR] agent.server: error performing anti-entropy sync of federation state: error="context canceled" 2021-04-16T16:15:03.501Z [ERROR] agent.server.raft: failed to decode incoming command: error="read tcp 10.27.52.42:8300->10.27.54.17:53925: read: connection timed out" 2021-04-16T16:15:48.690Z [ERROR] agent.server: error performing anti-entropy sync of federation state: error="context canceled" 2021-04-16T16:16:59.167Z [INFO] agent.server.serf.lan: serf: EventMemberJoin: consul-eap-ip-10-27-54-188-ip-10-27-54-188.us-west-2.compute.internal 10.27.54.188 2021-04-16T16:16:59.167Z [INFO] agent.server: Adding LAN server: server="consul-eap-ip-10-27-54-188-ip-10-27-54-188.us-west-2.compute.internal (Addr: tcp/10.27.54.188:8300) (DC: apiq-dev-us1-wei-eap)" 2021-04-16T16:16:59.168Z [INFO] agent.server.raft: updating configuration: command=AddNonvoter server-id=0bd86151-612d-47e6-a5c7-08287c3515e5 server-addr=10.27.54.188:8300 servers="[{Suffrage:Voter ID:659cafd8-5944-ecc0-39f4-9639d6f91395 Address:10.27.52.42:8300} {Suffrage:Voter ID:607ffdfd-64be-90b8-2ee1-f65c801aef7a Address:10.27.53.99:8300} {Suffrage:Nonvoter ID:0bd86151-612d-47e6-a5c7-08287c3515e5 Address:10.27.54.188:8300}]" 2021-04-16T16:16:59.170Z [INFO] agent.server.serf.wan: serf: EventMemberJoin: consul-eap-ip-10-27-54-188-ip-10-27-54-188.us-west-2.compute.internal.apiq-dev-us1-wei-eap 10.27.54.188 2021-04-16T16:16:59.170Z [INFO] agent.server: Handled event for server in area: event=member-join server=consul-eap-ip-10-27-54-188-ip-10-27-54-188.us-west-2.compute.internal.apiq-dev-us1-wei-eap area=wan 2021-04-16T16:16:59.170Z [INFO] agent.server.raft: added peer, starting replication: peer=0bd86151-612d-47e6-a5c7-08287c3515e5 2021-04-16T16:16:59.176Z [WARN] agent.server.raft: appendEntries rejected, sending older logs: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.188:8300}" next=245049 2021-04-16T16:16:59.178Z [INFO] agent.server: member joined, marking health alive: member=consul-eap-ip-10-27-54-188-ip-10-27-54-188.us-west-2.compute.internal 2021-04-16T16:16:59.178Z [INFO] agent.server.autopilot: Attempting removal of failed server node: id=0bd86151-612d-47e6-a5c7-08287c3515e5 name=consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal address=10.27.54.207:8300 2021-04-16T16:17:00.385Z [INFO] agent.server.raft: pipelining replication: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.188:8300}" 2021-04-16T16:17:07.316Z [INFO] agent.server.autopilot: Attempting removal of failed server node: id=0bd86151-612d-47e6-a5c7-08287c3515e5 name=consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal address=10.27.54.207:8300 2021-04-16T16:17:17.316Z [INFO] agent.server.autopilot: Attempting removal of failed server node: id=0bd86151-612d-47e6-a5c7-08287c3515e5 name=consul-eap-ip-10-27-54-17-ip-10-27-54-17.us-west-2.compute.internal address=10.27.54.17:8300 2021-04-16T16:17:27.316Z [INFO] agent.server.autopilot: Promoting server: id=0bd86151-612d-47e6-a5c7-08287c3515e5 address=10.27.54.188:8300 name=consul-eap-ip-10-27-54-188-ip-10-27-54-188.us-west-2.compute.internal 2021-04-16T16:17:27.316Z [INFO] agent.server.raft: updating configuration: command=AddStaging server-id=0bd86151-612d-47e6-a5c7-08287c3515e5 server-addr=10.27.54.188:8300 servers="[{Suffrage:Voter ID:659cafd8-5944-ecc0-39f4-9639d6f91395 Address:10.27.52.42:8300} {Suffrage:Voter ID:607ffdfd-64be-90b8-2ee1-f65c801aef7a Address:10.27.53.99:8300} {Suffrage:Voter ID:0bd86151-612d-47e6-a5c7-08287c3515e5 Address:10.27.54.188:8300}]" 2021-04-16T16:17:27.321Z [INFO] agent.server.autopilot: Attempting removal of failed server node: id=0bd86151-612d-47e6-a5c7-08287c3515e5 name=consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal address=10.27.54.207:8300 2021-04-16T16:17:37.313Z [INFO] agent.server.raft: updating configuration: command=RemoveServer server-id=0bd86151-612d-47e6-a5c7-08287c3515e5 server-addr= servers="[{Suffrage:Voter ID:659cafd8-5944-ecc0-39f4-9639d6f91395 Address:10.27.52.42:8300} {Suffrage:Voter ID:607ffdfd-64be-90b8-2ee1-f65c801aef7a Address:10.27.53.99:8300}]" 2021-04-16T16:17:37.314Z [INFO] agent.server.raft: removed peer, stopping replication: peer=0bd86151-612d-47e6-a5c7-08287c3515e5 last-index=245127 2021-04-16T16:17:37.315Z [INFO] agent.server.raft: aborting pipeline replication: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.188:8300}" 2021-04-16T16:17:37.317Z [INFO] agent.server.autopilot: removed server: id=0bd86151-612d-47e6-a5c7-08287c3515e5 2021-04-16T16:17:50.078Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.188:8300 leader=10.27.52.42:8300 2021-04-16T16:17:56.511Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.188:8300 leader=10.27.52.42:8300 2021-04-16T16:18:06.009Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.188:8300 leader=10.27.52.42:8300 2021-04-16T16:18:13.286Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.188:8300 leader=10.27.52.42:8300 2021-04-16T16:18:22.610Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.188:8300 leader=10.27.52.42:8300 2021-04-16T16:18:28.206Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.188:8300 leader=10.27.52.42:8300 2021-04-16T16:18:35.624Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.188:8300 leader=10.27.52.42:8300 2021-04-16T16:18:37.313Z [INFO] agent.server.raft: updating configuration: command=AddNonvoter server-id=0bd86151-612d-47e6-a5c7-08287c3515e5 server-addr=10.27.54.188:8300 servers="[{Suffrage:Voter ID:659cafd8-5944-ecc0-39f4-9639d6f91395 Address:10.27.52.42:8300} {Suffrage:Voter ID:607ffdfd-64be-90b8-2ee1-f65c801aef7a Address:10.27.53.99:8300} {Suffrage:Nonvoter ID:0bd86151-612d-47e6-a5c7-08287c3515e5 Address:10.27.54.188:8300}]" 2021-04-16T16:18:37.315Z [INFO] agent.server.raft: added peer, starting replication: peer=0bd86151-612d-47e6-a5c7-08287c3515e5 2021-04-16T16:18:37.316Z [ERROR] agent.server.raft: peer has newer term, stopping replication: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.188:8300}" 2021-04-16T16:18:37.316Z [INFO] agent.server.raft: entering follower state: follower="Node at 10.27.52.42:8300 [Follower]" leader= 2021-04-16T16:18:37.316Z [INFO] agent.server.raft: aborting pipeline replication: peer="{Voter 607ffdfd-64be-90b8-2ee1-f65c801aef7a 10.27.53.99:8300}" 2021-04-16T16:18:37.316Z [ERROR] agent.server.autopilot: failed to add raft non-voting peer: id=0bd86151-612d-47e6-a5c7-08287c3515e5 address=10.27.54.188:8300 error="leadership lost while committing log" 2021-04-16T16:18:37.316Z [ERROR] agent.server: failed to reconcile member: member="{consul-eap-ip-10-27-54-188-ip-10-27-54-188.us-west-2.compute.internal 10.27.54.188 8301 map[acls:0 build:1.9.3:f55da930 dc:apiq-dev-us1-wei-eap expect:3 ft_fs:1 ft_si:1 id:0bd86151-612d-47e6-a5c7-08287c3515e5 port:8300 raft_vsn:3 role:consul segment: use_tls:1 vsn:2 vsn_max:3 vsn_min:2 wan_join_port:8302] alive 1 5 2 2 5 4}" error="leadership lost while committing log" 2021-04-16T16:18:37.316Z [ERROR] agent.server.autopilot: failed to remove raft server: id=0bd86151-612d-47e6-a5c7-08287c3515e5 error="node is not the leader" 2021-04-16T16:18:37.316Z [ERROR] agent.server: failed to reconcile member: member="{consul-eap-ip-10-27-54-17-ip-10-27-54-17.us-west-2.compute.internal 10.27.54.17 8301 map[acls:0 build:1.9.3:f55da930 dc:apiq-dev-us1-wei-eap expect:3 ft_fs:1 ft_si:1 id:0bd86151-612d-47e6-a5c7-08287c3515e5 port:8300 raft_vsn:3 role:consul segment: use_tls:1 vsn:2 vsn_max:3 vsn_min:2 wan_join_port:8302] left 1 5 2 2 5 4}" error="node is not the leader" 2021-04-16T16:18:37.316Z [ERROR] agent.server.autopilot: failed to remove raft server: id=0bd86151-612d-47e6-a5c7-08287c3515e5 error="node is not the leader" 2021-04-16T16:18:37.316Z [ERROR] agent.server: failed to reconcile member: member="{consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal 10.27.54.207 8301 map[acls:0 build:1.9.3:f55da930 dc:apiq-dev-us1-wei-eap expect:3 ft_fs:1 ft_si:1 id:0bd86151-612d-47e6-a5c7-08287c3515e5 port:8300 raft_vsn:3 role:consul segment: use_tls:1 vsn:2 vsn_max:3 vsn_min:2 wan_join_port:8302] left 1 5 2 2 5 4}" error="node is not the leader" 2021-04-16T16:18:37.316Z [INFO] agent.server: cluster leadership lost 2021-04-16T16:18:41.491Z [WARN] agent.server.raft: rejecting vote request since our last index is greater: candidate=10.27.54.188:8300 last-index=245144 last-candidate-index=245127 2021-04-16T16:18:43.367Z [WARN] agent.server.raft: heartbeat timeout reached, starting election: last-leader= 2021-04-16T16:18:43.367Z [INFO] agent.server.raft: entering candidate state: node="Node at 10.27.52.42:8300 [Candidate]" term=49 2021-04-16T16:18:43.378Z [INFO] agent.server.raft: election won: tally=2 2021-04-16T16:18:43.378Z [INFO] agent.server.raft: entering leader state: leader="Node at 10.27.52.42:8300 [Leader]" 2021-04-16T16:18:43.378Z [INFO] agent.server.raft: added peer, starting replication: peer=607ffdfd-64be-90b8-2ee1-f65c801aef7a 2021-04-16T16:18:43.378Z [INFO] agent.server: cluster leadership acquired 2021-04-16T16:18:43.378Z [INFO] agent.server.raft: added peer, starting replication: peer=0bd86151-612d-47e6-a5c7-08287c3515e5 2021-04-16T16:18:43.378Z [INFO] agent.server: New leader elected: payload=consul-eap-ip-10-27-52-42-ip-10-27-52-42.us-west-2.compute.internal 2021-04-16T16:18:43.379Z [INFO] agent.server.raft: pipelining replication: peer="{Voter 607ffdfd-64be-90b8-2ee1-f65c801aef7a 10.27.53.99:8300}" 2021-04-16T16:18:43.381Z [WARN] agent.server.raft: appendEntries rejected, sending older logs: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.188:8300}" next=245128 2021-04-16T16:18:43.384Z [INFO] agent.server.raft: pipelining replication: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.188:8300}" 2021-04-16T16:18:43.386Z [INFO] agent.leader: started routine: routine="federation state anti-entropy" 2021-04-16T16:18:43.386Z [INFO] agent.leader: started routine: routine="federation state pruning" 2021-04-16T16:18:43.386Z [INFO] agent.leader: started routine: routine="intermediate cert renew watch" 2021-04-16T16:18:43.386Z [INFO] agent.leader: started routine: routine="CA root pruning" 2021-04-16T16:18:43.386Z [INFO] agent.server.raft: updating configuration: command=RemoveServer server-id=0bd86151-612d-47e6-a5c7-08287c3515e5 server-addr= servers="[{Suffrage:Voter ID:659cafd8-5944-ecc0-39f4-9639d6f91395 Address:10.27.52.42:8300} {Suffrage:Voter ID:607ffdfd-64be-90b8-2ee1-f65c801aef7a Address:10.27.53.99:8300}]" 2021-04-16T16:18:43.388Z [INFO] agent.server.raft: removed peer, stopping replication: peer=0bd86151-612d-47e6-a5c7-08287c3515e5 last-index=245147 2021-04-16T16:18:43.388Z [INFO] agent.server.raft: aborting pipeline replication: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.188:8300}" 2021-04-16T16:18:43.391Z [INFO] agent.server.autopilot: removed server: id=0bd86151-612d-47e6-a5c7-08287c3515e5 2021-04-16T16:18:43.391Z [INFO] agent.server.raft: updating configuration: command=AddNonvoter server-id=0bd86151-612d-47e6-a5c7-08287c3515e5 server-addr=10.27.54.188:8300 servers="[{Suffrage:Voter ID:659cafd8-5944-ecc0-39f4-9639d6f91395 Address:10.27.52.42:8300} {Suffrage:Voter ID:607ffdfd-64be-90b8-2ee1-f65c801aef7a Address:10.27.53.99:8300} {Suffrage:Nonvoter ID:0bd86151-612d-47e6-a5c7-08287c3515e5 Address:10.27.54.188:8300}]" 2021-04-16T16:18:43.393Z [INFO] agent.server.raft: added peer, starting replication: peer=0bd86151-612d-47e6-a5c7-08287c3515e5 2021-04-16T16:18:43.394Z [WARN] agent.server.raft: appendEntries rejected, sending older logs: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.188:8300}" next=245148 2021-04-16T16:18:43.396Z [INFO] agent.server.raft: pipelining replication: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.188:8300}" 2021-04-16T16:18:43.396Z [INFO] agent.server.autopilot: Attempting removal of failed server node: id=0bd86151-612d-47e6-a5c7-08287c3515e5 name=consul-eap-ip-10-27-54-17-ip-10-27-54-17.us-west-2.compute.internal address=10.27.54.17:8300 2021-04-16T16:19:03.386Z [INFO] agent.server.autopilot: Promoting server: id=0bd86151-612d-47e6-a5c7-08287c3515e5 address=10.27.54.188:8300 name=consul-eap-ip-10-27-54-188-ip-10-27-54-188.us-west-2.compute.internal 2021-04-16T16:19:03.386Z [INFO] agent.server.raft: updating configuration: command=AddStaging server-id=0bd86151-612d-47e6-a5c7-08287c3515e5 server-addr=10.27.54.188:8300 servers="[{Suffrage:Voter ID:659cafd8-5944-ecc0-39f4-9639d6f91395 Address:10.27.52.42:8300} {Suffrage:Voter ID:607ffdfd-64be-90b8-2ee1-f65c801aef7a Address:10.27.53.99:8300} {Suffrage:Voter ID:0bd86151-612d-47e6-a5c7-08287c3515e5 Address:10.27.54.188:8300}]" 2021-04-16T16:19:13.386Z [INFO] agent.server.autopilot: Attempting removal of failed server node: id=0bd86151-612d-47e6-a5c7-08287c3515e5 name=consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal address=10.27.54.207:8300 2021-04-16T16:19:23.386Z [INFO] agent.server.autopilot: Attempting removal of failed server node: id=0bd86151-612d-47e6-a5c7-08287c3515e5 name=consul-eap-ip-10-27-54-17-ip-10-27-54-17.us-west-2.compute.internal address=10.27.54.17:8300 2021-04-16T16:19:33.386Z [INFO] agent.server.autopilot: Attempting removal of failed server node: id=0bd86151-612d-47e6-a5c7-08287c3515e5 name=consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal address=10.27.54.207:8300 2021-04-16T16:19:43.382Z [INFO] agent.server.raft: updating configuration: command=RemoveServer server-id=0bd86151-612d-47e6-a5c7-08287c3515e5 server-addr= servers="[{Suffrage:Voter ID:659cafd8-5944-ecc0-39f4-9639d6f91395 Address:10.27.52.42:8300} {Suffrage:Voter ID:607ffdfd-64be-90b8-2ee1-f65c801aef7a Address:10.27.53.99:8300}]" 2021-04-16T16:19:43.384Z [INFO] agent.server.raft: removed peer, stopping replication: peer=0bd86151-612d-47e6-a5c7-08287c3515e5 last-index=245162 2021-04-16T16:19:43.384Z [INFO] agent.server.raft: aborting pipeline replication: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.188:8300}" 2021-04-16T16:19:43.386Z [INFO] agent.server.autopilot: removed server: id=0bd86151-612d-47e6-a5c7-08287c3515e5 2021-04-16T16:19:43.387Z [INFO] agent.server.raft: updating configuration: command=AddNonvoter server-id=0bd86151-612d-47e6-a5c7-08287c3515e5 server-addr=10.27.54.188:8300 servers="[{Suffrage:Voter ID:659cafd8-5944-ecc0-39f4-9639d6f91395 Address:10.27.52.42:8300} {Suffrage:Voter ID:607ffdfd-64be-90b8-2ee1-f65c801aef7a Address:10.27.53.99:8300} {Suffrage:Nonvoter ID:0bd86151-612d-47e6-a5c7-08287c3515e5 Address:10.27.54.188:8300}]" 2021-04-16T16:19:43.388Z [INFO] agent.server.raft: added peer, starting replication: peer=0bd86151-612d-47e6-a5c7-08287c3515e5 2021-04-16T16:19:43.389Z [WARN] agent.server.raft: appendEntries rejected, sending older logs: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.188:8300}" next=245163 2021-04-16T16:19:43.393Z [INFO] agent.server.raft: pipelining replication: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.188:8300}" 2021-04-16T16:20:03.386Z [INFO] agent.server.autopilot: Attempting removal of failed server node: id=0bd86151-612d-47e6-a5c7-08287c3515e5 name=consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal address=10.27.54.207:8300 2021-04-16T16:20:13.386Z [INFO] agent.server.autopilot: Attempting removal of failed server node: id=0bd86151-612d-47e6-a5c7-08287c3515e5 name=consul-eap-ip-10-27-54-17-ip-10-27-54-17.us-west-2.compute.internal address=10.27.54.17:8300 2021-04-16T16:20:23.386Z [INFO] agent.server.autopilot: Promoting server: id=0bd86151-612d-47e6-a5c7-08287c3515e5 address=10.27.54.188:8300 name=consul-eap-ip-10-27-54-188-ip-10-27-54-188.us-west-2.compute.internal 2021-04-16T16:20:23.386Z [INFO] agent.server.raft: updating configuration: command=AddStaging server-id=0bd86151-612d-47e6-a5c7-08287c3515e5 server-addr=10.27.54.188:8300 servers="[{Suffrage:Voter ID:659cafd8-5944-ecc0-39f4-9639d6f91395 Address:10.27.52.42:8300} {Suffrage:Voter ID:607ffdfd-64be-90b8-2ee1-f65c801aef7a Address:10.27.53.99:8300} {Suffrage:Voter ID:0bd86151-612d-47e6-a5c7-08287c3515e5 Address:10.27.54.188:8300}]" 2021-04-16T16:20:23.390Z [INFO] agent.server.autopilot: Attempting removal of failed server node: id=0bd86151-612d-47e6-a5c7-08287c3515e5 name=consul-eap-ip-10-27-54-207-ip-10-27-54-207.us-west-2.compute.internal address=10.27.54.207:8300 2021-04-16T16:20:43.383Z [INFO] agent.server.raft: updating configuration: command=RemoveServer server-id=0bd86151-612d-47e6-a5c7-08287c3515e5 server-addr= servers="[{Suffrage:Voter ID:659cafd8-5944-ecc0-39f4-9639d6f91395 Address:10.27.52.42:8300} {Suffrage:Voter ID:607ffdfd-64be-90b8-2ee1-f65c801aef7a Address:10.27.53.99:8300}]" 2021-04-16T16:20:43.385Z [INFO] agent.server.raft: removed peer, stopping replication: peer=0bd86151-612d-47e6-a5c7-08287c3515e5 last-index=245178 2021-04-16T16:20:43.385Z [INFO] agent.server.raft: aborting pipeline replication: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.188:8300}" 2021-04-16T16:20:43.388Z [INFO] agent.server.autopilot: removed server: id=0bd86151-612d-47e6-a5c7-08287c3515e5 2021-04-16T16:20:43.388Z [INFO] agent.server.raft: updating configuration: command=AddNonvoter server-id=0bd86151-612d-47e6-a5c7-08287c3515e5 server-addr=10.27.54.188:8300 servers="[{Suffrage:Voter ID:659cafd8-5944-ecc0-39f4-9639d6f91395 Address:10.27.52.42:8300} {Suffrage:Voter ID:607ffdfd-64be-90b8-2ee1-f65c801aef7a Address:10.27.53.99:8300} {Suffrage:Nonvoter ID:0bd86151-612d-47e6-a5c7-08287c3515e5 Address:10.27.54.188:8300}]" 2021-04-16T16:20:43.390Z [INFO] agent.server.raft: added peer, starting replication: peer=0bd86151-612d-47e6-a5c7-08287c3515e5 2021-04-16T16:20:43.391Z [WARN] agent.server.raft: appendEntries rejected, sending older logs: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.188:8300}" next=245179 2021-04-16T16:20:43.393Z [INFO] agent.server.raft: updating configuration: command=RemoveServer server-id=0bd86151-612d-47e6-a5c7-08287c3515e5 server-addr= servers="[{Suffrage:Voter ID:659cafd8-5944-ecc0-39f4-9639d6f91395 Address:10.27.52.42:8300} {Suffrage:Voter ID:607ffdfd-64be-90b8-2ee1-f65c801aef7a Address:10.27.53.99:8300}]" 2021-04-16T16:20:43.393Z [INFO] agent.server.raft: pipelining replication: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.188:8300}" 2021-04-16T16:20:43.395Z [INFO] agent.server.raft: removed peer, stopping replication: peer=0bd86151-612d-47e6-a5c7-08287c3515e5 last-index=245180 2021-04-16T16:20:43.395Z [INFO] agent.server.raft: aborting pipeline replication: peer="{Nonvoter 0bd86151-612d-47e6-a5c7-08287c3515e5 10.27.54.188:8300}" 2021-04-16T16:20:43.398Z [INFO] agent.server.autopilot: removed server: id=0bd86151-612d-47e6-a5c7-08287c3515e5 2021-04-16T16:20:54.880Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.188:8300 leader=10.27.52.42:8300 2021-04-16T16:21:00.553Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.188:8300 leader=10.27.52.42:8300 2021-04-16T16:21:07.289Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.188:8300 leader=10.27.52.42:8300 2021-04-16T16:21:14.269Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.188:8300 leader=10.27.52.42:8300 2021-04-16T16:21:21.251Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.188:8300 leader=10.27.52.42:8300 2021-04-16T16:21:28.374Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.188:8300 leader=10.27.52.42:8300 2021-04-16T16:21:35.969Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.188:8300 leader=10.27.52.42:8300 2021-04-16T16:21:41.685Z [WARN] agent.server.raft: rejecting vote request since we have a leader: from=10.27.54.188:8300 leader=10.27.52.42:8300 2021-04-16T16:21:43.387Z [INFO] agent.server.raft: updating configuration: command=AddNonvoter server-id=0bd86151-612d-47e6-a5c7-08287c3515e5 server-addr=10.27.54.188:8300 servers="[{Suffrage:Voter ID:659cafd8-5944-ecc0-39f4-9639d6f91395 Address:10.27.52.42:8300} {Suffrage:Voter ID:607ffdfd-64be-90b8-2ee1-f65c801aef7a Address:10.27.53.99:8300} {Suffrage:Nonvoter ID:0bd86151-612d-47e6-a5c7-08287c3515e5 Address:10.27.54.188:8300}]" 2021-04-16T16:21:43.392Z [INFO] agent.server.raft: added peer, starting replication: peer=0bd86151-612d-47e6-a5c7-08287c3515e5 2021-04-16T16:21:43.395Z [INFO] agent.server.raft: updating configuration: command=RemoveServer server-id=0bd86151-612d-47e6-a5c7-08287c3515e5 server-addr= servers="[{Suffrage:Voter ID:659c