I have a vault server with HA enabled & 2 replicas, consul storage and auto-unseal with Azure Keyvault.
Somehow my vault re-do this frequently on logs:
[WARN] core: leadership lost, stopping active operation
[INFO] core: pre-seal teardown starting
[INFO] rollback: stopping rollback manager
[INFO] core: pre-seal teardown complete
[WARN] core.cluster-listener: no TLS config found for ALPN: ALPN=["req_fw_sb-act_v1"]
[INFO] core: acquired lock, enabling active operation
[INFO] core: post-unseal setup starting
[INFO] core: loaded wrapping token key
[INFO] core: successfully setup plugin catalog: plugin-directory=""
[INFO] core: successfully mounted backend: type=system path=/
[INFO] core: successfully mounted backend: type=identity path=/
[INFO] core: successfully mounted backend: type=cubbyhole path=/
[INFO] core: successfully enabled credential backend: type=token path=/
[INFO] rollback: starting rollback manager
[INFO] core: restoring leases
[INFO] expiration: lease restore complete
[INFO] identity: entities restored
[INFO] identity: groups restored
[INFO] core: usage gauge collection is disabled
[INFO] core: post-unseal setup complete
[WARN] core: leadership lost, stopping active operation
...
One of Vault servers got 503, it maybe consul error or network, it’s no problem when vault can wait until the error gone. I tried making consul return 503 manually, Vault works as well, it waits untill Consul ready again.
[INFO] core: post-unseal setup complete
[WARN] core: leadership lost, stopping active operation
[INFO] core: pre-seal teardown starting
[INFO] rollback: stopping rollback manager
[INFO] core: pre-seal teardown complete
[ERROR] core: key rotation periodic upgrade check failed: error="Unexpected response code: 503"
[ERROR] core: failed to acquire lock: error="failed to read lock: Unexpected response code: 503"
[ERROR] core: key rotation periodic upgrade check failed: error="Unexpected response code: 503"
[ERROR] core: error during forwarded RPC request: ...
[ERROR] core: error during forwarded RPC request: ...
[ERROR] core: forward request error: error="error during forwarding RPC request"
[ERROR] core: forward request error: error="error during forwarding RPC request"
[INFO] core: acquired lock, enabling active operation
[INFO] core: post-unseal setup starting
[INFO] core: loaded wrapping token key
[INFO] core: successfully setup plugin catalog: plugin-directory=""
[INFO] core: successfully mounted backend: type=system path=/
[INFO] core: successfully mounted backend: type=identity path=/
[INFO] rollback: starting rollback manager
[INFO] core: restoring leases
[INFO] identity: entities restored
[INFO] expiration: lease restore complete
[INFO] identity: groups restored
[INFO] core: usage gauge collection is disabled
[INFO] core: post-unseal setup complete
But another one failed at post-unseal step, immediately Vault is sealed, there are no re-try or anymore logs.
[ERROR] core: post-unseal setup failed: error="<whatever message>: Unexpected response code: 503"
[INFO] core: stopping cluster listeners
[INFO] core.cluster-listener: forwarding rpc listeners stopped
[INFO] core.cluster-listener: rpc listeners successfully shut down
[INFO] core: cluster listeners successfully shut down
[INFO] core: vault is sealed
I got into this problem several times, only thing I can do is re-start Vault manual. Could someone please tell me is there anyway to make Vault auto unseal again? Or what could I do to prevent it happens?