Unable to disable a secret engine

Vault
1

Vault is unable to disable completely a secret engine, which is now stuck in the middle of not being usable anymore and not completely disabled.

  • Vault Server Version ): Version 1.15.6
  • Vault CLI Version : Vault v1.15.6 (615cf6f), built 2024-02-28T17:07:34Z
  • Server Operating System/Architecture: Debian GNU/Linux 12 (bookworm)

The vault server was version 1.14.10, and recently updated to version 1.15.6.
The disable initially was triggered via UI, and the secret engine had a significant number of records, with around 21499 pages in the search via UI.

Nothing with that secret engine works anymore, and the vault disable is failing constantly with:

Vault audit: delete failed - reason: 1 error occurred:

  • invalid request

And when trying to create a secret it says:

route entry is tainted.

We have tried the leases force removal but it also didn’t work, we didn’t have any errors while trying to do the lease revoke, the logs suggest that everything worked:

{"time":"2024-11-05T09:19:02.038441926Z","type":"request","auth":{"client_token":"hmac-sha256:0d0dbdc7e6d7c6db317cca7dfe3b3371e715bd21505a239046b2912252e68390","accessor":"hmac-sha256:994ad73e98749a0bcd6f48c88424c022df78d4358508bd4dde31c9480748931c","display_name":"root","policies":["root"],"token_policies":["root"],"policy_results":{"allowed":true,"granting_policies":[{"name":"root","namespace_id":"root","type":"acl"}]},"token_type":"service","token_issue_time":"2020-05-06T08:08:39Z"},"request":{"id":"bb04a459-73e9-4e83-1a84-e16e0e7c2d11","client_id":"0DHqvq2D77kL2/JTPSZkTMJbkFVmUu0TzMi0jiXcFy8=","operation":"update","mount_point":"sys/","mount_type":"system","mount_accessor":"system_7e1ce005","mount_running_version":"v1.15.6+builtin.vault","mount_class":"secret","client_token":"hmac-sha256:0d0dbdc7e6d7c6db317cca7dfe3b3371e715bd21505a239046b2912252e68390","client_token_accessor":"hmac-sha256:994ad73e98749a0bcd6f48c88424c022df78d4358508bd4dde31c9480748931c","namespace":{"id":"root"},"path":"sys/leases/revoke-force/opg/data","remote_address":"127.0.0.1","remote_port":22898}}

Vault audit: update succesfull

The following two commands were issued, and both returned as successful:

vault lease revoke -force -prefix opg/data/
Warning! Force-removing leases can cause Vault to become out of sync with
secret engines!
Success! Force revoked any leases with prefix: opg/data/

and

vault lease revoke -force -prefix sys/mounts/opg/data/
Warning! Force-removing leases can cause Vault to become out of sync with
secret engines!
Success! Force revoked any leases with prefix: sys/mounts/opg/data/

Details can be found here: Secret engine not being disabled · Issue #28682 · hashicorp/vault · GitHub

2

Extra failure we noticed recently, is that everytime we try to access some other secret engines, it faills, it’s weird that it’s faling at sys/license/status now:


Vault audit: read failed - reason: 1 error occurred:
    * unsupported path
{"time":"2024-12-02T07:50:52.611489806Z","type":"response","auth":{"client_token":"hmac-sha256:0d0dbdc7e6d7c6db317cca7dfe3b3371e715bd21505a239046b2912252e68390","accessor":"hmac-sha256:994ad73e98749a0bcd6f48c88424c022df78d4358508bd4dde31c9480748931c","display_name":"root","policies":["root"],"token_policies":["root"],"policy_results":{"allowed":true,"granting_policies":[{"name":"root","namespace_id":"root","type":"acl"}]},"token_type":"service","token_issue_time":"2020-05-06T08:08:39Z"},"request":{"id":"fcf36bf3-721e-1d02-263f-9cc9fb2b8c88","client_id":"0DHqvq2D77kL2/JTPSZkTMJbkFVmUu0TzMi0jiXcFy8=","operation":"read","mount_point":"sys/","mount_type":"system","mount_accessor":"system_7e1ce005","mount_running_version":"v1.15.6+builtin.vault","mount_class":"secret","client_token":"hmac-sha256:0d0dbdc7e6d7c6db317cca7dfe3b3371e715bd21505a239046b2912252e68390","client_token_accessor":"hmac-sha256:994ad73e98749a0bcd6f48c88424c022df78d4358508bd4dde31c9480748931c","namespace":{"id":"root"},"path":"sys/license/status","remote_address":"127.0.0.1","remote_port":51182},"response":{"mount_point":"sys/","mount_type":"system","mount_accessor":"system_7e1ce005","mount_running_plugin_version":"v1.15.6+builtin.vault","mount_class":"secret","data":{"error":"hmac-sha256:85b73df8c1c1ec52f9bf941ceec14f022736032878ca717b8d5c6886d7a3859b"}},"error":"1 error occurred:\n\t* unsupported path\n\n"}