I am upgrading our neglected vault implementation from 0.6.5 to latest (1.3.2), changing the backend and implementing auto unseal in the process.
I have upgrade incrementally, reading the release notesof each version, up to latest without any noticeable issue.
Then I have implemented aws/kms autounseal and I have migrated the backend from zookeeper to dynamodb.
Up to this point all went well. Then I started seeing a huge amount of errors related to vaiult being unable to revoke leases on DBs that do not exist anymore.
020-02-19T17:23:24.422Z [ERROR] expiration: failed to revoke lease: lease_id=mongo-r9/creds/readWrite/7b16aa59-826f-12db-fd90-fce9d650cfff error="failed to revoke entry: resp: (*logical.Response)(nil) err: no reachable servers"
This brought a short DOS to vault, as the configured throughput of Dynamodb was reached.
ProvisionedThroughputExceededException: The level of configured provisioned throughput for the table was exceeded. Consider increasing your provisioning level with the UpdateTable API.
At this point, I have googled around for ways to disable an unreachable secrets backend and I stumbled on this:
This approach worked for some secrets engines, but it fails constantly on others:
bash-5.0# vault lease revoke -f -prefix mongo-r9 Warning! Force-removing leases can cause Vault to become out of sync with secret engines! Error force revoking leases with prefix mongo-r9: context deadline exceeded
I am at loss on how to procede from here, any help is appreciated