Error migrating to raft on new VM

My goal is to create a new Vault VM with the content from an existing Vault instance. The existing Vault uses the file system backend, the new CentOS 8 VM will use the Integrated Storage (raft) backend running Vault 1.8.1. The migration succeeds and /opt/vault/data on the new VM contains the following:

drwxr-xr-x. 3 vault vault 4096 Aug 17 08:52 raft
-rw-------. 1 vault vault 6160384 Aug 17 09:01 vault.db

/opt/vault/data/raft contains the following:

-rw-------. 1 vault vault 4612096 Aug 17 09:01 raft.db
drwxr-xr-x. 2 vault vault 4096 Aug 17 08:52 snapshots

vault status indicates that Vault is uninitialized; vault operator init returns:

Error initializing: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/sys/init
Code: 400. Errors:

  • could not bootstrap clustered storage: error bootstrapping cluster: cluster already has state

I also appended log records below.

My plan is to initially run raft in a single node cluster to enable online backups. I’m starting with TLS disabled until I get things working.

Any thoughts in how to get running are appreciated.

Thanks,
Rich


systemctl status vault

● vault.service - “HashiCorp Vault - A tool for managing secrets”
Loaded: loaded (/usr/lib/systemd/system/vault.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2021-08-17 09:01:40 CDT; 40min ago
Docs: Documentation | Vault | HashiCorp Developer
Main PID: 40730 (vault)
Tasks: 9 (limit: 49540)
Memory: 20.7M
CGroup: /system.slice/vault.service
└─40730 /usr/bin/vault server -config=/etc/vault.d/vault.hcl

Aug 17 09:01:40 tapis-vault-stage vault[40730]: Version: Vault v1.8.1
Aug 17 09:01:40 tapis-vault-stage vault[40730]: Version Sha: 4b0264f28defc05454c31277cfa6ff63695a458d
Aug 17 09:01:40 tapis-vault-stage vault[40730]: ==> Vault server started! Log data will stream in below:
Aug 17 09:01:40 tapis-vault-stage vault[40730]: 2021-08-17T09:01:40.594-0500 [INFO] proxy environment: http_proxy=“” https_proxy=“” no_proxy=“”
Aug 17 09:03:54 tapis-vault-stage vault[40730]: 2021-08-17T09:03:54.145-0500 [INFO] core: security barrier not initialized
Aug 17 09:03:54 tapis-vault-stage vault[40730]: 2021-08-17T09:03:54.145-0500 [INFO] core: seal configuration missing, not initialized
Aug 17 09:05:27 tapis-vault-stage vault[40730]: 2021-08-17T09:05:27.546-0500 [INFO] core: security barrier not initialized
Aug 17 09:05:27 tapis-vault-stage vault[40730]: 2021-08-17T09:05:27.546-0500 [ERROR] core: failed to bootstrap raft: error="could not bootstrap clustered storage: error bootstra>
Aug 17 09:42:02 tapis-vault-stage vault[40730]: 2021-08-17T09:42:02.421-0500 [INFO] core: security barrier not initialized
Aug 17 09:42:02 tapis-vault-stage vault[40730]: 2021-08-17T09:42:02.422-0500 [INFO] core: seal configuration missing, not initialized

What did the migrate file look like?
Was it run on the current Vault node or the new Vault VM?

Mike, thanks for taking a look at this. The existing Vault instance has been running in Kubernetes for more than a year. I stopped that instance, rsync-ed its data over to a new VM outside of Kubernetes, and then ran the migration configuration below. The migration indicated it completed successfully.

storage_source “file” {
path = “/home/guest/stagevault”
}

storage_destination “raft” {
path = “/opt/vault/data”
node_id = “raft_node_1”
}

cluster_addr = “http://127.0.0.1:8201

What does vault operator raft list-peers show?
In your vault.hcl config, what does your storage stanza look like?

My storage config looked like this:

storage “raft” {
path = “/opt/vault/data”
node_id = “raft_node_1”
}
cluster_addr = “http://127.0.0.1:8201

I say “looked” because I’ve given up on migration. I’ve decided instead to start with a fresh Vault installation and rebuild my content from scratch. I’ll have to add roles, policies and secrets that I programmatically extract from the existing system. Not the way I hoped to go, but an approach that should get me to my goal.

Thanks for your help.

Good time to point out using the terraform Vault provider can make this task alot easier…