Overview of the Issue
A new consul cluster (v 0.8.1) was built to replace current Consul cluster (v 0.8.1). Took the consul snap backup from current cluster and restored it onto the new cluster. During the post-unseal process, the following was observed :
[DEBUG] core: shutting down leader elections
[DEBUG] core: finished triggering standbyStopCh for runStandby
[DEBUG] core: runStandby done
[DEBUG] core: sealing barrier
[INFO] core: vault is sealed
[INFO] core: vault is unsealed
[INFO] core: entering standby mode
[INFO] core: acquired lock, enabling active operation
[DEBUG] core: generating cluster private key
[DEBUG] core: generating local cluster certificate
[INFO] core: post-unseal setup starting
[DEBUG] core: clearing forwarding clients
[DEBUG] core: done clearing forwarding clients
[INFO] core: loaded wrapping token key
[INFO] core: successfully setup plugin catalog: plugin-directory=
[INFO] core: successfully mounted backend: type=generic path=secret/
[INFO] core: successfully mounted backend: type=system path=sys/
[INFO] core: successfully mounted backend: type=identity path=identi ty/
[INFO] core: successfully mounted backend: type=pki path=pki/
[INFO] core: successfully mounted backend: type=cubbyhole path=cubby hole/
[INFO] core: successfully mounted backend: type=pki path=my/ldap/pki/
[ERROR] core: failed to mount entry: path=my/ldap/pki/ error="cannot mount under existing mount "my/ldap/pki/""
[INFO] core: pre-seal teardown starting
[INFO] core: cluster listeners not running
[INFO] core: pre-seal teardown complete
It then goes into a loop and remains in status 429 (standby) :
$ consul operator raft list-peers
Node ID Address State Voter
my-consul-node1 x.x.x.x:8300 x.x.x.x:8300 leader true
my-consul-node2 x.x.x.x:8300 x.x.x.x:8300 follower true
my-consul-node3 x.x.x.x:8300 x.x.x.x:8300 follower true
$ vault status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 1
Threshold 1
Version 1.0.3
Cluster Name vault-cluster-624c7ebf
Cluster ID 8e2ac018-3d9e-1e84-e4da-76146b54bf83
HA Enabled true
HA Cluster n/a
HA Mode standby
Active Node Address <none>
The current cluster does not show a duplication of mounts :
$ vault secrets list
Path Type Accessor Description
---- ---- -------- -----------
cubbyhole/ cubbyhole cubbyhole_295119ab per-token private secret storage
identity/ identity identity_275d4e5e identity store
pki/ pki pki_e71d366f n/a
secret/ generic generic_2c3dc747 generic secret storage
my/ldap/pki/ pki pki_97dff7b7 n/a
my/ldap2/pki/ pki pki_fd1cbf54 n/a
sys/ system system_71e16012 system endpoints used for control, policy and debugging
A side note : I also tried building a new cluster with the latest consul version (1.7.2) and restoring the snapshot with the exact same steps as described below, and I received exactly the same issue. I am doing this activity in preparation of upgrading our current Vault/Consul infrastructure (which is very old!) to the latest versions.
Reproduction Steps
Steps to reproduce this issue, eg:
- Create a new cluster with the same version of consul (0.8.1) and vault (1.0.3)
- Initialize and unseal new cluster with the new keys
- Get a copy of consul snap backup from current cluster (consul.snap.2021-06-15_1200), onto the new cluster
- Run a snapshot restore:
consul snapshot restore consul.snap.2021-06-15_1200 - Delete core lock
consul kv delete vault/core/lock - Unseal the vault with the current cluster’s master key:
vault operator unseal - Watch the log as the mounting operation fails
Consul info for both Client and Server
agent:
check_monitors = 0
check_ttls = 1
checks = 1
services = 2
build:
prerelease =
revision = 'e9ca44d
version = 0.8.1
consul:
bootstrap = false
known_datacenters = 1
leader = false
leader_addr = x.x.x.x:8300
server = true
raft:
applied_index = 20661759
commit_index = 20661759
fsm_pending = 0
last_contact = 5.359637ms
last_log_index = 20661760
last_log_term = 2
last_snapshot_index = 20658044
last_snapshot_term = 2
latest_configuration = [{Suffrage:Voter ID:x.x.x.x:8300 Address:x.x.x.x:8300} {Suffrage:Voter ID:x.x.x.x:8300 Address:x.x.x.x:8300} {Suffrage:Voter ID:x.x.x.x:8300 Address:x.x.x.x:8300}]
latest_configuration_index = 1
num_peers = 2
protocol_version = 2
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 2
runtime:
arch = amd64
cpu_count = 2
goroutines = 72
max_procs = 2
os = linux
version = go1.8.1
serf_lan:
encrypted = true
event_queue = 0
event_time = 2
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 3
members = 3
query_queue = 0
query_time = 2
serf_wan:
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 8
members = 3
query_queue = 0
query_time = 2
Operating system and Environment details
RHEL 8
