There is a little Consul cluster migrated to new network.
Consider old net = 192.168.1.0/24, new net = 10.20.30.0/24.
There are three Vault nodes configured as cluster using Consul backend:
disable_mlock = true
cluster_name = "test1"
max_lease_ttl = "8760h"
default_lease_ttl = "8760h"
listener "tcp" {
address = "10.20.30.41:8200"
tls_cert_file = "/etc/vault/tls/vault.crt"
tls_key_file = "/etc/vault/tls/vault.key"
tls_min_version = "tls12"
tls_cipher_suites = "..."
tls_prefer_server_cipher_suites = "false"
tls_disable = "false"
}
backend "consul" {
address = "127.0.0.1:8500"
datacenter = "test1"
path = "vault"
service = "vault"
cluster_addr = "https://10.20.30.41:8201"
redirect_addr = "https://10.20.30.41:8200"
disable_clustering = "False"
scheme = "http"
}
ui = true
Problem:
Vault standby nodes (running on 10.20.30.41 and 10.20.30.42) are trying to access active node (actually running in 10.20.30.43) using old IP (192.168.1.43):
Server status:
# vault status -address https://10.20.30.41:8200 -tls-skip-verify Key Value --- ----- Seal Type shamir Initialized true Sealed false Total Shares 5 Threshold 3 Version 0.11.2 Cluster Name test1 Cluster ID d0cc1d1e-e0fd-601d-7ebc-7b90a00482ae HA Enabled true HA Cluster https://192.168.1.43:8201 HA Mode standby Active Node Address https://192.168.1.43:8200
Server output:
==> Vault server configuration:
Api Address: https://10.20.30.41:8200
Cgo: disabled
Cluster Address: https://10.20.30.41:8201
Listener 1: tcp (addr: "10.20.30.41:8200", cluster address: "10.20.30.41:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")
Log Level: debug
Mlock: supported: true, enabled: false
Storage: consul (HA available)
Version: Vault v0.11.2
Version Sha: 2b1a4304374712953ff606c6a925bbe90a4e85dd
==> Vault server started! Log data will stream in below:
[DEBUG] storage.consul: config path set: path=vault
[WARN ] storage.consul: appending trailing forward slash to path
[DEBUG] storage.consul: config disable_registration set: disable_registration=false
[DEBUG] storage.consul: config service set: service=vault
[DEBUG] storage.consul: config service_tags set: service_tags=
[DEBUG] storage.consul: config service_address set: service_address=
[DEBUG] storage.consul: config address set: address=127.0.0.1:8500
[DEBUG] storage.consul: config scheme set: scheme=http
[DEBUG] storage.cache: creating LRU cache: size=0
[DEBUG] cluster listener addresses synthesized: cluster_addresses=[10.20.30.41:8201]
[DEBUG] core: cannot unseal, not enough keys: keys=1 threshold=3 nonce=9284c56c-1585-10d6-5118-47962adea603
[DEBUG] core: cannot unseal, not enough keys: keys=2 threshold=3 nonce=9284c56c-1585-10d6-5118-47962adea603
[INFO ] core: vault is unsealed
[INFO ] core: entering standby mode
[DEBUG] core: parsing information for new active node:
active_cluster_addr=https://192.168.1.43:8201
active_redirect_addr=https://192.168.1.43:8200
[DEBUG] core: refreshing forwarding connection
[DEBUG] core: clearing forwarding clients
[DEBUG] core: done clearing forwarding clients
[DEBUG] core: done refreshing forwarding connection
Question:
How to forget old active_cluster_addr and active_redirect_addr, and rejoin to active node using new ipaddr?