Cannot Re-Initialise Vault after rebuilding Consul without a backup

Vault
1

I’ve recently rebuilt consul without taking a backup of Vault’s data (not very familiar with either Consul or Vault unfortunately).

Consul is back up and running and the configuration for Consul & Vault hasn’t changed.

I’ve tried to reinitialise Vault from scratch, however, I’m presented with the following error message “GET https://<vault_server_address>/v1/sys/seal-status/:EOF”

I’ve also tried to use Vault with raft storage to try and get around the problem, however, when I try to re-initialise Vault again, the same error is thrown.

Please can someone help advise what can be done to re-initialise Vault successfully again?

Many thanks

2

When you say you are trying to reinitialize Vault, do you mean set up a new, empty cluster? Or bring the cluster back online that was previously running (presumably using the Consul cluster you rebuilt as a backend/storage layer)?

3

I was planning to bring back online the cluster that was previously running since everything is already in place, if possible.

4

If you don’t have a Vault backup to restore, and the previous Consul backend is not available anymore, I don’t think that will be an option.

Do you have any previous Vault backups prior to rebuilding Consul?

5

Oh right, I see.

I’m afraid I don’t have any previous Vault backups for this particular environment unfortunately.

6

If you’re new to Vault, we have some good learning resources you can follow for free on developer.hashicorp.com.

Check out this collection as a starting point:

When preparing to build your new Vault cluster, these may be helpful:

And documentation on backups is here (click the how-to guides tab)

Hopefully these can help you get up and running with a new cluster.

Another option, depending on your use case is to use Vault on the HashiCorp Cloud Platform - things like upgrades and backups are managed there and gives you support options.

7

Thank you very much indeed for your help! Much appreciated :slight_smile:

Will checkout the links you kindly provided.

Would the existing configuration including the api server address still work on the newly deployed cluster?

8

As long as no other system/service is using that address, that should all be fine.

1 Like
9

That’s great news, thank you! Will delete the old cluster and spin up new machines.

Thanks so much for your help :slight_smile:

1 Like
10

Hi @jonathanfrappier

We’re trying to delete the old vault nodes from the infrastructure and deploy new ones as you kindly suggested, however, terraform still queries Vault for different things during the deployment. Since Vault has been down for quite some time now, the info it’s looking for is not there and Vault is also inaccessible.

If the terraform plan is approved and we go ahead with the deployment, this is likely to fail because of the Vault errors.

I was wondering if there is a way to stop terraform from trying to connect to Vault during the deployment at all? Would removing the vault provider from terraform help with this?

Many thanks in advance :slight_smile:

11

This depends on how you/your team use Terraform and how it was utilizing Vault. Without understanding your setup, this is a best guess set of steps so please consider these with caution as I dont want to guide you into a breaking change.

You probably should get any references to the Vault resources out of your Terraform state:

Once its out of your state, then you can remove the vault items from the terraform configuration.

1 Like
12

That makes sense - thank you very much indeed for your help @jonathanfrappier :slight_smile:

13

Happy to help! Good luck with getting Vault back up and everything working smoothly again.