My consul cluster is dead, I would like to know how to recover/migrate consul key value data from this dead cluster(no cluster leader) to new one?
I have 2 questions
1, how to resume this dead consul leader(no cluster leader)
2, is it possible to directly copy data folder from this dead consul cluster to another new consul cluster to recover somehow? thanks.
My current consul cluster’s error log:
server0
47:8300}" error=“dial tcp ->10.42.4.147:8300: i/o timeout”
12/30/2021 5:47:30 PM 2021-12-30T09:47:30.877Z [ERROR] agent.anti_entropy: failed to sync remote state: error=“No cluster leader”
12/30/2021 5:47:31 PM 2021-12-30T09:47:31.166Z [INFO] agent.server.raft: duplicate requestVote for same term: term=34769
12/30/2021 5:47:33 PM 2021-12-30T09:47:33.348Z [WARN] agent.server.raft: Election timeout reached, restarting election
12/30/2021 5:47:33 PM 2021-12-30T09:47:33.348Z [INFO] agent.server.raft: entering candidate state: node=“Node at 10.42.4.149:8300 [Candidate]” term=34770
12/30/2021 5:47:33 PM 2021-12-30T09:47:33.419Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=8bc26ffd-89f8-7b11-984d-f2834448ca2c fallback=10.42.4.147:8300 error=“Could not find address for server id 8bc26ffd-89f8-7b11-984d-f2834448ca2c”
12/30/2021 5:47:38 PM 2021-12-30T09:47:38.309Z [ERROR] agent.server.raft: failed to make requestVote RPC: target="{Voter 8bc26ffd-89f8-7b11-984d-f2834448ca2c 10.42.4.147:8300}" error=“dial tcp ->10.42.4.147:8300: i/o timeout”
12/30/2021 5:47:38 PM 2021-12-30T09:47:38.802Z [INFO] agent.server.raft: duplicate requestVote for same term: term=34770
12/30/2021 5:47:40 PM 2021-12-30T09:47:40.410Z [WARN] agent.server.raft: Election timeout reached, restarting election
12/30/2021 5:47:40 PM 2021-12-30T09:47:40.410Z [INFO] agent.server.raft: entering candidate state: node=“Node at 10.42.4.149:8300 [Candidate]” term=34771
12/30/2021 5:47:40 PM 2021-12-30T09:47:40.415Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=8bc26ffd-89f8-7b11-984d-f2834448ca2c fallback=10.42.4.147:8300 error=“Could not find address for server id 8bc26ffd-89f8-7b11-984d-f2834448ca2c”
12/30/2021 5:47:43 PM 2021-12-30T09:47:43.419Z [ERROR] agent.server.raft: failed to make requestVote RPC: target="{Voter 8bc26ffd-89f8-7b11-984d-f2834448ca2c 10.42.4.147:8300}" error=“dial tcp ->10.42.4.147:8300: i/o timeout”
12/30/2021 5:47:45 PM 2021-12-30T09:47:45.336Z [INFO] agent.server.raft: duplicate requestVote for same term: term=34771
12/30/2021 5:47:47 PM 2021-12-30T09:47:47.622Z [ERROR] agent: Coordinate update error: error=“No cluster leader”
12/30/2021 5:47:50 PM 2021-12-30T09:47:50.239Z [WARN] agent.server.raft: Election timeout reached, restarting election
12/30/2021 5:47:50 PM 2021-12-30T09:47:50.239Z [INFO] agent.server.raft: entering candidate state: node=“Node at 10.42.4.149:8300 [Candidate]” term=34772
12/30/2021 5:47:50 PM 2021-12-30T09:47:50.244Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=8bc26ffd-89f8-7b11-984d-f2834448ca2c fallback=10.42.4.147:8300 error=“Could not find address for server id 8bc26ffd-89f8-7b11-984d-f2834448ca2c”
12/30/2021 5:47:50 PM 2021-12-30T09:47:50.415Z [ERROR] agent.server.raft: failed to make requestVote RPC: target="{Voter 8bc26ffd-89f8-7b11-984d-f2834448ca2c 10.42.4.147:8300}" error=“dial tcp ->10.42.4.147:8300: i/o timeout”
server1
2021-12-30T09:47:56.896Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=8bc26ffd-89f8-7b11-984d-f2834448ca2c fallback=10.42.4.147:8300 error=“Could not find address for server id 8bc26ffd-89f8-7b11-984d-f2834448ca2c”
12/30/2021 5:48:00 PM 2021-12-30T09:48:00.781Z [ERROR] agent.server.raft: failed to make requestVote RPC: target="{Voter 8bc26ffd-89f8-7b11-984d-f2834448ca2c 10.42.4.147:8300}" error=“dial tcp ->10.42.4.147:8300: i/o timeout”
12/30/2021 5:48:06 PM 2021-12-30T09:48:06.698Z [WARN] agent.server.raft: Election timeout reached, restarting election
12/30/2021 5:48:06 PM 2021-12-30T09:48:06.698Z [INFO] agent.server.raft: entering candidate state: node=“Node at 10.42.5.172:8300 [Candidate]” term=34774
12/30/2021 5:48:06 PM 2021-12-30T09:48:06.721Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=8bc26ffd-89f8-7b11-984d-f2834448ca2c fallback=10.42.4.147:8300 error=“Could not find address for server id 8bc26ffd-89f8-7b11-984d-f2834448ca2c”
12/30/2021 5:48:06 PM 2021-12-30T09:48:06.896Z [ERROR] agent.server.raft: failed to make requestVote RPC: target="{Voter 8bc26ffd-89f8-7b11-984d-f2834448ca2c 10.42.4.147:8300}" error=“dial tcp ->10.42.4.147:8300: i/o timeout”
12/30/2021 5:48:14 PM 2021-12-30T09:48:14.074Z [WARN] agent.server.raft: Election timeout reached, restarting election
12/30/2021 5:48:14 PM 2021-12-30T09:48:14.075Z [INFO] agent.server.raft: entering candidate state: node=“Node at 10.42.5.172:8300 [Candidate]” term=34775
12/30/2021 5:48:14 PM 2021-12-30T09:48:14.080Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=8bc26ffd-89f8-7b11-984d-f2834448ca2c fallback=10.42.4.147:8300 error=“Could not find address for server id 8bc26ffd-89f8-7b11-984d-f2834448ca2c”
12/30/2021 5:48:16 PM 2021-12-30T09:48:16.250Z [ERROR] agent: Coordinate update error: error=“No cluster leader”
12/30/2021 5:48:16 PM 2021-12-30T09:48:16.721Z [ERROR] agent.server.raft: failed to make requestVote RPC: target="{Voter 8bc26ffd-89f8-7b11-984d-f2834448ca2c 10.42.4.147:8300}" error=“dial tcp ->10.42.4.147:8300: i/o timeout”
12/30/2021 5:48:23 PM 2021-12-30T09:48:23.279Z [WARN] agent.server.raft: Election timeout reached, restarting election
12/30/2021 5:48:23 PM 2021-12-30T09:48:23.279Z [INFO] agent.server.raft: entering candidate state: node=“Node at 10.42.5.172:8300 [Candidate]” term=34776
12/30/2021 5:48:23 PM 2021-12-30T09:48:23.294Z [WARN] agent.server.raft: unable to get address for sever, using fallback address: id=8bc26ffd-89f8-7b11-984d-f2834448ca2c fallback=10.42.4.147:8300 error=“Could not find address for server id 8bc26ffd-89f8-7b11-984d-f2834448ca2c”
12/30/2021 5:48:24 PM 2021-12-30T09:48:24.080Z [ERROR] agent.server.raft: failed to make requestVote RPC: target="{Voter 8bc26ffd-89f8-7b11-984d-f2834448ca2c 10.42.4.147:8300}" error=“dial tcp ->10.42.4.147:8300: i/o timeout”