Restore from Backup - "check-and-set parameter did not match the current version"

Vault: v1.9.3
O.S.: Centos7
HA = Enabled
storage_type = Consul v1.11.2

Hi,

I’m testing Vault with Consul for a production rollout. I did a backup (consul snapshot save), went into the UI and added a new kv pair, and then restored the backup–this was to test basic functionality. The data I had added last disappeared as expected; however, when I went to re-add it, I receive the following error, which is persistent.

Error
check-and-set parameter did not match the current version

By contrast, I am able to add new data.

Any idea what might be going on? Could this be a bug or am I approaching backup/restore wrong?

Thanks!

UPDATE: I can retrieve the old secret using the CLI, but I can’t see it in the UI. Unclear if it survived the restore or Vault re-added it despite throwing that error. Very odd!

UPDATE AGAIN I see this from the documentation for kv delete:

The kv delete command deletes the data for the provided path in the key/value secrets engine. If using K/V Version 2, its versioned data will not be fully removed, but marked as deleted and will no longer be returned in normal get requests.

I didn’t run that command, but I wonder if I have created a similar effect by running the restore process in this way against a K/V version 2 secrets engine.

Did you happen to do the restore while Vault was running?