Am running a consul cluster on EKS which i would call it as primary datacenter(dc1) and one of the EC2 machine has consul as well, named it as dc2. I am trying to join dc1 and dc2 but getting the below errors
2021-10-20T14:08:28.604Z [WARN] agent.fsm: EnsureRegistration failed: error="failed inserting node: Error while renaming Node ID: "6b2683be-5348-639b-fa9e-ee28a4928f92": Node name ip-10-0-137-163.us-east-2.compute.internal is reserved by node 11db113a-08cd-8fae-8493-8f3d2e6197ea with name ip-10-0-137-163.us-east-2.compute.internal (10.0.142.51)"
2021-10-20T14:08:32.838Z [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 10.0.143.18:8302: read tcp 10.0.31.21:50188->10.0.13.89:8443: read: connection reset by peer
2021-10-20T14:08:35.191Z [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 10.0.143.18:8302: dial tcp <nil>->10.0.138.181:8443: connect: no route to host
2021-10-20T14:08:36.764Z [WARN] agent.server.rpc: RPC request to DC is currently failing as no server can be reached: datacenter=dc2
2021-10-20T14:08:39.790Z [WARN] agent.fsm: EnsureRegistration failed: error="failed inserting node: Error while renaming Node ID: "32036f22-4a98-72a4-9a9a-1e343839391c": Node name ip-10-0-11-179.us-east-2.compute.internal is reserved by node 68cbc6ba-c9b9-7352-51eb-989558a5e145 with name ip-10-0-11-179.us-east-2.compute.internal (10.0.25.125)"
I could telnet from the K8 consul servers to VM and the other way around, also, the mesh gateway of my K8 cluster is a classic loadbalancer.