Cannot delete old secrets engine

Vault
1

I am upgrading our neglected vault implementation from 0.6.5 to latest (1.3.2), changing the backend and implementing auto unseal in the process.

I have upgrade incrementally, reading the release notesof each version, up to latest without any noticeable issue.
Then I have implemented aws/kms autounseal and I have migrated the backend from zookeeper to dynamodb.

Up to this point all went well. Then I started seeing a huge amount of errors related to vaiult being unable to revoke leases on DBs that do not exist anymore.

020-02-19T17:23:24.422Z [ERROR] expiration: failed to revoke lease: lease_id=mongo-r9/creds/readWrite/7b16aa59-826f-12db-fd90-fce9d650cfff error="failed to revoke entry: resp: (*logical.Response)(nil) err: no reachable servers"

This brought a short DOS to vault, as the configured throughput of Dynamodb was reached.

ProvisionedThroughputExceededException: The level of configured provisioned throughput for the table was exceeded. Consider increasing your provisioning level with the UpdateTable API.

At this point, I have googled around for ways to disable an unreachable secrets backend and I stumbled on this:

This approach worked for some secrets engines, but it fails constantly on others:

bash-5.0# vault lease revoke -f -prefix mongo-r9
Warning! Force-removing leases can cause Vault to become out of sync with secret engines!
Error force revoking leases with prefix mongo-r9: context deadline exceeded

I am at loss on how to procede from here, any help is appreciated

2

After watching the logs closely I found that the revoked leases weren’t the same for each run. So, while it was going in timeout before revoking all leases, the leases were being revoked.

After realizing this I have quickly solved the issue by putting the lease revoke command in a loop for 20 times. After that the secret engines could be disabled without issues.

3

I am having a similar issue. What is the command you used for lease revoke?

4

Nvm, this did it for me vault lease revoke -f -prefix path/