TL;DR: While attempting to migrate vault’s backend to dynamodb, I’ve discovered that a consul secrets engine I was using for testing still exists in vault, but the associated consul server, and all of its ACLs, tokens, and leases is long gone.
Vault will not let me disable/delete this consul engine with error messages saying it can’t reach the non-existent consul server. It is trying to revoke what it believes are outstanding tokens/leases. I am unable to reconfigure this engine, getting a No value found at consul-dctest/config/access, despite the engine path in vault secrets list showing as consul-dctest.
It should also be noted that vault read consul/config/access works properly to read out the production configuration. The point being that AFAICT vault is behaving properly except for this specific consul-dctest engine.
I’ve also tried writing new config to consul-dctest/config/access and get an error 404 no handler for route 'consul-dctest/config/access'
I tried setting up an empty consul server at the address vault is trying to talk to when I say vault secrets disable consul-dctest - hoping that would get past the issue, but vault fails to disable the engine, trying to revoke leases -
* failed to revoke "consul-dctest/creds/agent-role/<random_string>" (1 / 33): failed to revoke entry: resp: (*logical.Response)(nil) err: Unexpected response code: 403 (ACL not found)
As noted above, vault also isn’t letting me do any reconfiguration of the engine to set/reset any consul ACL tokens. I’m stuck trying to figure out how to delete/disable this consul-dctest engine. Any ideas? thanks!