Vault Unseal - Code: 500 cipher: message authentication failed


I’ve noticed a few threads / questions around issues with the vault unseal and a code 500. My understanding is this essentially means out unseal key is incorrect, but I am just trying to understand what could have caused this to occur.

We run a vault instance in Kubernetes and use ansible to deploy and setup the vault instance.

We store the vault key as a kubernetes secret and keep a backup (encrypted) copy of the original init message on the system in case we ever get outselves into a situation where we need it.

Recently we had a server outage and after getting everything backup and running we’re now finding that the vault instance is stuck and unable to unseal. Checking the init message it looks to be un-modified since Sept of last year, so our unseal key should still be valid, however we’re getting the 500 code as described above.

I’m just trying to understand what . how the vault instance could have gotten out of sync and what we could potentially do to recover it (though that seems like it might not be possible) and future proof against this happening.