There is a little Consul cluster migrated to new network.
Consider old net = 192.168.1.0/24, new net = 10.20.30.0/24.
There are three Vault nodes configured as cluster using Consul backend:
disable_mlock = true cluster_name = "test1" max_lease_ttl = "8760h" default_lease_ttl = "8760h" listener "tcp" { address = "10.20.30.41:8200" tls_cert_file = "/etc/vault/tls/vault.crt" tls_key_file = "/etc/vault/tls/vault.key" tls_min_version = "tls12" tls_cipher_suites = "..." tls_prefer_server_cipher_suites = "false" tls_disable = "false" } backend "consul" { address = "127.0.0.1:8500" datacenter = "test1" path = "vault" service = "vault" cluster_addr = "https://10.20.30.41:8201" redirect_addr = "https://10.20.30.41:8200" disable_clustering = "False" scheme = "http" } ui = true
Problem:
Vault standby nodes (running on 10.20.30.41 and 10.20.30.42) are trying to access active node (actually running in 10.20.30.43) using old IP (192.168.1.43):
Server status:
# vault status -address https://10.20.30.41:8200 -tls-skip-verify Key Value --- ----- Seal Type shamir Initialized true Sealed false Total Shares 5 Threshold 3 Version 0.11.2 Cluster Name test1 Cluster ID d0cc1d1e-e0fd-601d-7ebc-7b90a00482ae HA Enabled true HA Cluster https://192.168.1.43:8201 HA Mode standby Active Node Address https://192.168.1.43:8200
Server output:
==> Vault server configuration: Api Address: https://10.20.30.41:8200 Cgo: disabled Cluster Address: https://10.20.30.41:8201 Listener 1: tcp (addr: "10.20.30.41:8200", cluster address: "10.20.30.41:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled") Log Level: debug Mlock: supported: true, enabled: false Storage: consul (HA available) Version: Vault v0.11.2 Version Sha: 2b1a4304374712953ff606c6a925bbe90a4e85dd ==> Vault server started! Log data will stream in below: [DEBUG] storage.consul: config path set: path=vault [WARN ] storage.consul: appending trailing forward slash to path [DEBUG] storage.consul: config disable_registration set: disable_registration=false [DEBUG] storage.consul: config service set: service=vault [DEBUG] storage.consul: config service_tags set: service_tags= [DEBUG] storage.consul: config service_address set: service_address= [DEBUG] storage.consul: config address set: address=127.0.0.1:8500 [DEBUG] storage.consul: config scheme set: scheme=http [DEBUG] storage.cache: creating LRU cache: size=0 [DEBUG] cluster listener addresses synthesized: cluster_addresses=[10.20.30.41:8201] [DEBUG] core: cannot unseal, not enough keys: keys=1 threshold=3 nonce=9284c56c-1585-10d6-5118-47962adea603 [DEBUG] core: cannot unseal, not enough keys: keys=2 threshold=3 nonce=9284c56c-1585-10d6-5118-47962adea603 [INFO ] core: vault is unsealed [INFO ] core: entering standby mode [DEBUG] core: parsing information for new active node: active_cluster_addr=https://192.168.1.43:8201 active_redirect_addr=https://192.168.1.43:8200 [DEBUG] core: refreshing forwarding connection [DEBUG] core: clearing forwarding clients [DEBUG] core: done clearing forwarding clients [DEBUG] core: done refreshing forwarding connection
Question:
How to forget old active_cluster_addr and active_redirect_addr, and rejoin to active node using new ipaddr?