Vault Lazarus Upgrade

I have inherited management of several Vault servers still running v0.x releases that I’m trying to upgrade to the latest. Still, I am running into issues when I attempt to start Vault after upgrading. The error I’m seeing is after starting vault and attempting to unseal. I’m receiving an error that says failed to setup mount table on the console, looking in the logs I see [ERROR] core: failed to mount entry path=cubbyhole/ error"cannot mount under existing \"cubbyhole/\"" but a few lines above I see where it says it successfully mounted the secret/, cubbyhole/ and sys/ backends so it appears like it is attempting to mount the cubbyhole plugin twice though before upgrading if I ran vault secrets list it only has cubbyhole, secret and sys listed. This is trying to upgrade from v0.6.x … I have another instance that is running v0.11.x and it is able to upgrade. The only difference I can find is that it has the identity/ mount that the v0.6.x instance doesn’t.

What is the backend storage you’re using? That’s probably more important in upgrading older versions than anything else. You have to pick the right backend, then start reading the upgrade guides for each of the versions between 0.6 and 0.11 to see how you can do the upgrade and what versions would work and not.

If this was me, and if the environment isn’t complicated I’d be writing a script to dump everything out, and start a new cluster and importing it in. A lot less work and a lot less chance of missing something since you can keep both up for a time to make sure you didn’t miss anything.

It is two standalone (non-clustered) Vault instances just using the filesystem backend. They are running 0.6.2, and I’ve been able to test and verify I can upgrade them to 0.6.5, which was the last 0.6.x release, but then trying to upgrade to 0.7.0 starts showing the cubbyhole mount error. I have another that is running 0.11.1 that I’ve been able to test upgrading to the current version without an issue configured like the other two.

Dumping and standing up a new Vault instance would be an issue affecting more than just my duties as all tokens would need to be replaced as well.

I guess I would add this to the win column that I found yet another 0.6.2 instance running. When I attempted to upgrade it to 1.11.2 I didn’t encounter the problem I was having. I had been chalking it up to the version, so was surprised by this. I went and tested the third 0.6.2 instance I hadn’t tested upgrading after the first failed, and it managed to upgrade without problem as well. So now I’m left wondering what is up with this one instance and how I can potentially salvage it and upgrade without having to start fresh.

All three were configured identically but on different servers. All are using the standard filesystem backend storage. Yet only the one appears to be having this issue with the cubbyhole/ default mount.