We are trying to restore a backup into a new cluster, for testing. Both clusters use the same version, 1.8.7. Auto-unseal is not enabled. We create a new cluster, initialize vault, restore the consul snapshot, then attempt to unseal the vault servers. The unseal process succeeds on the first server, but running “vault status” immediately afterword returns “sealed: true”. We have tried this numerous times - initialize vault first, restore backup first, backups from consul-backinator, backups from “consul snapshot” - yet we always get the same result. Something must be different but we have no idea what it is.
Are you sure it’s actually unsealing? Check your system logs. Most likely scenario is that your keys are actually not getting accepted after the last entry. Vault doesn’t actually check the validity of the keys you enter until all of them are entered.
The other less likely scenario is that Vault is somehow detecting a major issue with your backend and its trying to protect itself – this scenario would be clearly logged within the system logs and it’ll tell you why Vault thinks it needs to seal itself.
I finally got my co-worker to check the logs when this happens. We found that there is a custom plugin in use that needed to be brought over to the new cluster. It was crashing post-unseal because the plugin could not be loaded. Thanks!