[Solved] Delete (disable) consul engine fails

TL;DR: While attempting to migrate vault’s backend to dynamodb, I’ve discovered that a consul secrets engine I was using for testing still exists in vault, but the associated consul server, and all of its ACLs, tokens, and leases is long gone.

Vault will not let me disable/delete this consul engine with error messages saying it can’t reach the non-existent consul server. It is trying to revoke what it believes are outstanding tokens/leases. I am unable to reconfigure this engine, getting a No value found at consul-dctest/config/access, despite the engine path in vault secrets list showing as consul-dctest.

It should also be noted that vault read consul/config/access works properly to read out the production configuration. The point being that AFAICT vault is behaving properly except for this specific consul-dctest engine.

I’ve also tried writing new config to consul-dctest/config/access and get an error 404 no handler for route 'consul-dctest/config/access'

I tried setting up an empty consul server at the address vault is trying to talk to when I say vault secrets disable consul-dctest - hoping that would get past the issue, but vault fails to disable the engine, trying to revoke leases -

* failed to revoke "consul-dctest/creds/agent-role/<random_string>" (1 / 33): failed to revoke entry: resp: (*logical.Response)(nil) err: Unexpected response code: 403 (ACL not found)

As noted above, vault also isn’t letting me do any reconfiguration of the engine to set/reset any consul ACL tokens. I’m stuck trying to figure out how to delete/disable this consul-dctest engine. Any ideas? thanks!

Figures. After a day of messing around trying to fix this, give up trying because nothing is working. Post question. Go back to trying to solve problem and quickly find solution.

In any case, the trick was to force vault to forget about the leases, like so:

vault lease revoke -force -sync -prefix consul-dctest/creds/agent-role

After that completed, I was able to drop the consul-dctest engine.

ref: lease revoke - Command | Vault by HashiCorp

1 Like