we’re using multiple Vault servers with PostgreSQL as the storage backend and encountered a problem where all transit key versions created before the Vault service restart were not lost as we can still see the original number of key versions, but are not useable anymore to decrypt the previously encrypted content.
The failed decryption error message is
Code: 400. Errors:* cipher: message authentication failed
New versions/rotated of the transit key do work as expected after the restart.
This service uptime was multiple months before we restarted it, if that is helpful in any way.
As we first suspected auto rotation or the minimum decrypt version, but this is a redacted output of the transit key:
vault read transit/keys/flux-secrets
keys map[1:1658327089 10:1658364792 …]
We are also unable to reproduce this in QA.
After creating over 6000 transit key versions and then restarting the Vault service, we can still use the original v1 version of the transit key to decrypt.
Maybe there’s a time component I’m missing here.
Any help would be greatly appreciated.