Best practice to migrate members between datacenters?

Consul
1

I have two consul datacenters: vpnA and vpnB. Both have a bunch of leaders that can communicate with leaders of both datacenters. Members of different datacenters can’t communicate.

I’m now trying to move members from vpnA to vpnB, is this the correct approach?

  1. Stop consul, which causes the member to leave the consul cluster
  2. Purge the member from vpnA datacenter (consul force-leave -prune ...)
  3. Reconfigure the member, dropping vpnA IP and adding vpnB IP, point consul config to vpnB leaders and datacenter, advertising vpnB IP
  4. Remove the consul data dir
  5. Start consul

If I do so, the member proplery joins the vpnB datacenter cluster, but gets the following error and advertises no services:

[ERROR] agent.client: RPC failed to server: method=Coordinate.Update server=<vpnB-leader-ip>:8300 error="rpc error getting client: failed to get conn: dial tcp <vpnA-member-ip>:0-><vpnB-leader-ip>:8300: connect: network is unreachable"

I don’t really understand where it gets the old IP from. Any idea?

2

This has resolved itself, the problem was due to a misconfiguration of the member.