How to recover from "error removing server with duplicate ID"?

I’m using Consul and Nomad, running a single node cluster for both Consul and Nomad 1.3.2, and having issues with two separate Nomad servers instances appearing in Consul even though I’m only running one server

In blue is the actual IP of the server while red is an IP I don’t recognize. Also in the Nomad logs I’m seeing

 nomad: failed to reconcile: error="error removing server with duplicate ID \"f2d46f9f-ddc4-285a-a0d6-b3f77115c037\": need at least one voter in configuration: {[]}"

while checking raft returns

$ nomad operator raft list-peers
Node       ID                                    Address         State     Voter  RaftProtocol
(unknown)  f2d46f9f-ddc4-285a-a0d6-b3f77115c037  follower  true   unknown

How do I recover from this and remove the duplicate/unknown server instance? Other Nomad client nodes are also having trouble joining the Nomad cluster because I believe it’s trying to join the unknown server. Thanks for your help.

I was able to partially solve this by generating a new raft.peers file with

NOMAD_DATA_DIR=$(nomad agent-info -json | jq -r '.config.DataDir')
NOMAD_ADDR=$(nomad agent-info -json | jq -r '.stats.nomad.leader_addr')
NODE_ID=$(cat "$NOMAD_DATA_DIR/server/node-id")

cat <<EOF > "$NOMAD_DATA_DIR/server/raft/peers.json"
    "id": "$NODE_ID",
    "address": "$NOMAD_ADDR",
    "non_voter": false

and restarting the server. Now the client can join just fine and nomad operator raft list-peers returns appropriate response. However, I’m still seeing two Nomad server entries within Consul. How can I debug why it’s showing 2 when I only have 1 server?


Bumping this since I’m still having this issue, thanks