Unable to migrate k8s HA from etcd to raft while retaining S3 storage

Vault
1

I’m trying to work out how to migrate my HA storage from etcd to raft, while keeping data storage in S3; essentially I’d like to go from this configuration:

        api_addr = "https://127.0.0.1:8202"
        cluster_addr = "https://127.0.0.1:8201"
        disable_mlock = true
        service_registration "kubernetes" {}
        storage "s3" {
          bucket = "myawsaccountprefix-vault"
          region = "us-west-2"
        }
        listener "tcp" {
          address = "[::]:8200"
          cluster_address = "[::]:8201"
          tls_disable = true
        }
        seal "awskms" {
          region = "us-west-2"
          kms_key_id = "alias/myawsaccountprefix-vault"
        }
        ha_storage "etcd" {
          address = "http://vault-etcd.vault.svc.cluster.local:2379"
          ha_enabled = "true"
          etcd_api = "v3"
        }

to this configuration:

        api_addr = "https://127.0.0.1:8202"
        cluster_addr = "https://127.0.0.1:8201"
        disable_mlock = true
        service_registration "kubernetes" {}
        storage "s3" {
          bucket = "myawsaccountprefix-vault"
          region = "us-west-2"
        }
        listener "tcp" {
          address = "[::]:8200"
          cluster_address = "[::]:8201"
          tls_disable = true
        }
        seal "awskms" {
          region = "us-west-2"
          kms_key_id = "alias/myawsaccountprefix-vault"
        }
        ha_storage "raft" {
          path = "/vault/data/raft"
        }

however, I cannot successfully get the cluster to assign an active node after running vault operator migrate -config /vault/data/raft/migrate.hcl with the following migration config file:

$ cat /vault/data/raft/migrate.hcl
storage_source "etcd" {
  address = "http://vault-etcd.vault.svc.cluster.local:2379"
  ha_enabled = "true"
  etcd_api = "v3"
}
storage_destination "raft" {
  path = "/vault/data/raft"
}
api_addr = "https://127.0.0.1:8202"
cluster_addr = "https://127.0.0.1:8201"

when I apply the ha_storage "raft" config, all three nodes remain in standby mode; if I switch back to ha_storage "etcd" a node immediately becomes active.

I cannot find any examples of HA being coordinated by raft while data storage is external to the cluster (S3 in my case) but I see evidence ample evidence in the code that it’s an expected configuration, eg vault/vault/raft.go at main · hashicorp/vault · GitHub

2

I asked internally and confirmed your findings that this should work. If you’re running Vault community, I would suggest posting this as a Github issue. If you are using Vault Enterprise, please open a support ticket.

3

Thanks, will do!

Github Issue: Cannot migrate from etcd HA to raft HA when using external S3 storage · Issue #29259 · hashicorp/vault · GitHub

1 Like