Vault is sealed after: post-unseal setup failed

I have a vault server with HA enabled & 2 replicas, consul storage and auto-unseal with Azure Keyvault.
Somehow my vault re-do this frequently on logs:

[WARN] core: leadership lost, stopping active operation
[INFO] core: pre-seal teardown starting
[INFO] rollback: stopping rollback manager
[INFO] core: pre-seal teardown complete
[WARN] core.cluster-listener: no TLS config found for ALPN: ALPN=["req_fw_sb-act_v1"]
[INFO] core: acquired lock, enabling active operation
[INFO] core: post-unseal setup starting
[INFO] core: loaded wrapping token key
[INFO] core: successfully setup plugin catalog: plugin-directory=""
[INFO] core: successfully mounted backend: type=system path=/
[INFO] core: successfully mounted backend: type=identity path=/
[INFO] core: successfully mounted backend: type=cubbyhole path=/
[INFO] core: successfully enabled credential backend: type=token path=/
[INFO] rollback: starting rollback manager
[INFO] core: restoring leases
[INFO] expiration: lease restore complete
[INFO] identity: entities restored
[INFO] identity: groups restored
[INFO] core: usage gauge collection is disabled
[INFO] core: post-unseal setup complete
[WARN] core: leadership lost, stopping active operation
...

One of Vault servers got 503, it maybe consul error or network, it’s no problem when vault can wait until the error gone. I tried making consul return 503 manually, Vault works as well, it waits untill Consul ready again.

[INFO] core: post-unseal setup complete
[WARN] core: leadership lost, stopping active operation
[INFO] core: pre-seal teardown starting
[INFO] rollback: stopping rollback manager
[INFO] core: pre-seal teardown complete
[ERROR] core: key rotation periodic upgrade check failed: error="Unexpected response code: 503"
[ERROR] core: failed to acquire lock: error="failed to read lock: Unexpected response code: 503"
[ERROR] core: key rotation periodic upgrade check failed: error="Unexpected response code: 503"
[ERROR] core: error during forwarded RPC request: ...
[ERROR] core: error during forwarded RPC request: ...
[ERROR] core: forward request error: error="error during forwarding RPC request"
[ERROR] core: forward request error: error="error during forwarding RPC request"
[INFO] core: acquired lock, enabling active operation
[INFO] core: post-unseal setup starting
[INFO] core: loaded wrapping token key
[INFO] core: successfully setup plugin catalog: plugin-directory=""
[INFO] core: successfully mounted backend: type=system path=/
[INFO] core: successfully mounted backend: type=identity path=/
[INFO] rollback: starting rollback manager
[INFO] core: restoring leases
[INFO] identity: entities restored
[INFO] expiration: lease restore complete
[INFO] identity: groups restored
[INFO] core: usage gauge collection is disabled
[INFO] core: post-unseal setup complete 

But another one failed at post-unseal step, immediately Vault is sealed, there are no re-try or anymore logs.

[ERROR] core: post-unseal setup failed: error="<whatever message>: Unexpected response code: 503"
[INFO] core: stopping cluster listeners
[INFO] core.cluster-listener: forwarding rpc listeners stopped
[INFO] core.cluster-listener: rpc listeners successfully shut down
[INFO] core: cluster listeners successfully shut down
[INFO] core: vault is sealed

I got into this problem several times, only thing I can do is re-start Vault manual. Could someone please tell me is there anyway to make Vault auto unseal again? Or what could I do to prevent it happens?

Vault is very sensitive to both network and disk latency. My guess is that one of those is out of the expected range for Vault and that’s triggering the seal. It’s hard to be sure without so little information but that’s the first theory that I can come up with.

My suggestion is don’t screw around with it and open a case with support. They can walk through the myriad of different paths to figure out what is going on.

If you don’t already turn up your logging to DEBUG [vault and consul] and catch the event, the first thing they’re going to ask for is the debug logs from the servers (all of primary nodes).