3 server Vault cluster with Local Raft storage

Vault
1

Hi everyone, I’m completely new in Vault world so I apologize if I ask some dumb questions.
I wat to setup a Vault cluster with 3 nodes with local Raft storage. I am following the documentation as much as I can but I am not quite getting there in terms of joining second node to the Active node as a Standby.

On my Node 1 I have the following config for Vault

disable_mlock="true"

storage "raft" {
  path    = "/home/myUser/vault/data"
  node_id = "vault-server-1"

  retry_join {
    leader_api_addr = "https://vault-server-1:8200"
    leader_client_cert_file = "/home/myUser/etc/vault-server-1.crt"
    leader_client_key_file = "/home/myUser/etc/vault-server-1.private.key"
  }
  retry_join {
    leader_api_addr = "https://vault-server-2:8200"
    leader_client_cert_file = "/home/myUser/etc/vault-server-1.crt"
    leader_client_key_file = "/home/myUser/etc/vault-server-1.private.key"
  }
  retry_join {
    leader_api_addr = "https://vault-server-1:8200"
    leader_client_cert_file = "/home/myUser/etc/vault-server-1.crt"
    leader_client_key_file = "/home/myUser/etc/vault-server-1.private.key"
  }

}

seal "transit" {
  address     = "http://vault-server-0:8200"
  token       = "s.XXXXXXXXXXXXXXXXXXXXX"
  key_name    = "unseal-key"
  mount_path  = "transit"
}

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_cert_file = "/home/myUser/etc/vault-server-1.crt"
  tls_key_file  = "/home/myUser/etc/vault-server-1.private.key"
}

api_addr = "https://vault-server-1:8200"
cluster_addr = "https://vault-server-1:8201"
ui = true

I start the vault as per normal and I have the following status:

Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    5
Threshold                3
Version                  1.9.3
Storage Type             raft
Cluster Name             vault-cluster-e11d29d8
Cluster ID               efed540d-9ae5-a071-7a52-8679847446cb
HA Enabled               true
HA Cluster               https://vault-server-1:8201
HA Mode                  active
Active Since             2022-03-22T17:43:58.458859304Z
Raft Committed Index     353
Raft Applied Index       353

In the secondary node I have same config only node_id, api_addr and cluster_addr are respectively vault-server-2.

and vault status is as follows:

Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    5
Threshold                3
Version                  1.9.3
Storage Type             raft
Cluster Name             vault-cluster-2cc97a0c
Cluster ID               042cd5bd-5256-4f78-ea1a-35dafacbc068
HA Enabled               true
HA Cluster               https://vault-server-2:8201
HA Mode                  active
Active Since             2022-03-30T11:23:36.359872379Z
Raft Committed Index     86
Raft Applied Index       86

So both are in Active state, so how do I add server 2 as a standby for Active server 1 ?
As I understand I need to run something like this on server 2 ?

vault operator raft join https://vault-server-1:8200

it gives me the following result:

Key       Value
---       -----
Joined    true

But when I check the following on server 2 it’s still a leader:

~ > vault operator raft list-peers
Node                  Address                    State     Voter
----                  -------                    -----     -----
vault-server-2    vault-server-2:8201    leader    true

What am I doing wrong ?
Have to say in most of examples it’s recommended to use Consul, but I want to keep it more simple with Raft…
Please any advise ?

2

I’m still learning RAFT myself so take this with a grain of salt, but I think you may need to add a cluster_address parameter to your tcp listener block.

Also, if you can, specify the actual IP address of the instance in the tcp listener block. I believe there is/was some odd behavior if that is left up to Vault/Go to determine using the interface address.

3

Hi, thanks I have put the cluster_address in tcp listener block but it’s still the same…
Just to give you more details, so I have initialized separately both servers and I have the unseal keys and token, but then I used another vault server to auto-unseal which seems to be working when I start both vault instances, I wonder if I need to do something different with cluster join when I use auto-unseal ?

4

By the naming of the instances I’m going to assume you’re deploying into a kubernetes cluster? If so, I’d check the names of your pods to make sure they are resolvable, i.e. vault-server-2 can reach vault-server-1:8200 as well as https://vault-server-1:8201 via curl from inside of node 2. Most likely they cannot see/talk to each other, otherwise the retry_join – btw your posted config has the wrong retry-join for your 3rd node – would auto join them. You shouldn’t need to manually join a node unless you’re dynamically adding it to the cluster after the fact.

Just to note all the kub-names I have seen are pod-#.domain-internal:8200 … I’m a beginner Kubernetes so I maybe wrong but that’s what led me to this.

Re: Storage – raft vs consul does not matter here. You would run into the same issue using consul if the nodes can’t find each other.

Last note I’ll add is that I have this in my values.yaml – see if it helps (it shouldn’t as autopilot doesn’t come into effect until after the cluster is up and running.

  ha:
    enabled: true
    replicas: 5
    raft: 
      enabled: true 
      setNodeId: true 

    config: |
      ui = true
       
      listener "tcp" {
        address          = "[::]:8200"
        cluster_address  = "[::]:8201"
        tls_ca_cert_file = "/vault/userconfig/tls-ca/ca.crt"
        tls_cert_file    = "/vault/userconfig/tls-server/tls.crt"
        tls_key_file     = "/vault/userconfig/tls-server/tls.key"
      }

      storage "consul" {
        address = "consul.basement.lab:8500"
        path = "k8-vault/"

        retry_join { 
          leader_api_addr = "https://vault-0.vault-internal:8200"
          leader_ca_cert_file = "/vault/userconfig/tls-server/ca.crt"
          leader_client_cert_file = "/vault/userconfig/tls-server/server.crt"
          leader_client_key_file  = "/vault/userconfig/tls-server/server.key"
        }
        retry_join { 
          leader_api_addr = "https://vault-1.vault-internal:8200"
          leader_ca_cert_file = "/vault/userconfig/tls-server/ca.crt"
          leader_client_cert_file = "/vault/userconfig/tls-server/server.crt"
          leader_client_key_file  = "/vault/userconfig/tls-server/server.key"
        }
        retry_join { 
          leader_api_addr = "https://vault-2.vault-internal:8200"
          leader_ca_cert_file = "/vault/userconfig/tls-server/ca.crt"
          leader_client_cert_file = "/vault/userconfig/tls-server/server.crt"
          leader_client_key_file  = "/vault/userconfig/tls-server/server.key"
        }
        retry_join { 
          leader_api_addr = "https://vault-3.vault-internal:8200"
          leader_ca_cert_file = "/vault/userconfig/tls-server/ca.crt"
          leader_client_cert_file = "/vault/userconfig/tls-server/server.crt"
          leader_client_key_file  = "/vault/userconfig/tls-server/server.key"
        }
        retry_join { 
          leader_api_addr = "https://vault-4.vault-internal:8200"
          leader_ca_cert_file = "/vault/userconfig/tls-server/ca.crt"
          leader_client_cert_file = "/vault/userconfig/tls-server/server.crt"
          leader_client_key_file  = "/vault/userconfig/tls-server/server.key"
        }

        autopilot {
          cleanup_dead_servers = "true"
          last_contact_threshold = "200ms"
          last_contact_failure_threshold = "10m"
          max_trailing_logs = 250000
          min_quorum = 5
          server_stabilization_time = "10s"
        }
      }

      service_registration "kubernetes" {}
5

This is where you’re making a misstep - initialize is an operation you do once per cluster, not once per node. As a result, you now have two separate single node clusters:

and

From this state, these two nodes are never going to join together as nodes of a common cluster, as this would involve throwing away one set of data.

Give it a try initialising only one node.

2 Likes
7

Thanks a lot I was thinking that this is my mistake too but wasn’t sure.
Btw I am not on Kubernetes , I have Red Had servers on VM.
I will retry with initializing just one node and let you know if it helps…

8

I’ve just put together a 3 node cluster on Red Hat VMs (actually I am using docker on them) so, I thought, I will share my config as it may help.

Notes:

  • this is the config on the first node. Each node has it’s own domain name (vault01.test.example.com, vault02.test.example.com and vault03.test.example.com)
  • the cluster_addr and raft.node_id will change on each node AND the retry_join stanzas are always listing the “other” two nodes
  • we got a load balancer in front which has vault-test.example.com domain name assigned and it forwards the request to all nodes (non active nodes will forward the requests to the active node)
  • I created a single tls cert which has all the domain names (vault01.test, vault02.test, vault03.test, vault-test) in subjectAltName and I use the same cert to all to simplify
  • we use AWS multi region KMS key for auto-unseal
cluster_name                        = "vault-test"
cluster_addr                        = "https://vault01.test.example.com:8201"
api_addr                            = "https://vault-test.example.com"

# recommended for internal storage
disable_mlock                       = true

# enable ui so we can look around :)
ui                                  = true

# Enables the addition of an HTTP header in all of Vault's HTTP responses: X-Vault-Raft-Node-ID
enable_response_header_raft_node_id = true

# Trace, Debug, Error, Warn, Info
log_level     = "Info"

listener "tcp" {
  address            = "0.0.0.0:8200"
  tls_client_ca_file = "/vault/config/tls/ca.crt"
  tls_cert_file      = "/vault/config/tls/tls.crt"
  tls_key_file       = "/vault/config/tls/tls.key"
}

seal "awskms" {
  region     = "REMOVED"
  access_key = "REMOVED"
  secret_key = "REMOVED"
  kms_key_id = "arn:aws:kms:REMOVED:REMOVED:key/REMOVED"
}

storage "raft" {
  path    = "/vault/raft"
  node_id = "vault01"

  retry_join {
    leader_tls_servername   = "vault02.test.example.com"
    leader_api_addr         = "https://vault02.test.example.com:8200"
    leader_ca_cert_file     = "/vault/config/tls/ca.crt"
    leader_client_cert_file = "/vault/config/tls/tls.crt"
    leader_client_key_file  = "/vault/config/tls/tls.key"
  }
  retry_join {
    leader_tls_servername   = "vault03.test.example.com"
    leader_api_addr         = "https://vault03.test.example.com:8200"
    leader_ca_cert_file     = "/vault/config/tls/ca.crt"
    leader_client_cert_file = "/vault/config/tls/tls.crt"
    leader_client_key_file  = "/vault/config/tls/tls.key"
  }

}

telemetry {
  prometheus_retention_time = "24h"
  disable_hostname = true
}
1 Like
9

@maxb @Cajga
Thanks a lot guys I have sorted the issue with your advise.
First of all I initialized only 1st instance and then just started the rest.
I had to add leader_ca_cert_file in the config as it was complaining during joining process with below:

join attempt failed: error="error during raft bootstrap init call: Put \/v1/sys/storage/raft/bootstrap/challenge\": x509: certificate signed by unknown authority"

Now it looks good

 > vault operator raft list-peers
Node                Address                State       Voter
----                -------                -----       -----
vault-server-1		vault-server-1:8201    leader      true
vault-server-2   	vault-server-2:8201    follower    true
vault-server-3    	vault-server-3:8201    follower    true

My current config for node 1 looks like this:

disable_mlock="true"

storage "raft" {
  path    = "/home/MyUser/vault/data"
  node_id = "vault-server-1"

  retry_join {
    leader_tls_servername   = "vault-server-2"
    leader_api_addr = "https://vault-server-2:8200"
    leader_ca_cert_file = "/home/MyUser/etc/vault-server-chain.pem"
    leader_client_cert_file = "/home/MyUser/etc/vault-server.crt"
    leader_client_key_file = "/home/MyUser/etc/vault-server.private.key"
  }
  retry_join {
    leader_tls_servername   = "vault-server-3"
    leader_api_addr = "https://vault-server-3:8200"
    leader_ca_cert_file = "/home/MyUser/etc/vault-server-chain.pem"
    leader_client_cert_file = "/home/MyUser/etc/vault-server.crt"
    leader_client_key_file = "/home/MyUser/etc/vault-server.private.key"
  }

}

seal "transit" {
  address     = "http://vault-server-0:8200"
  token       = "s.XXXXXXXXX"
  key_name    = "unseal-key"
  mount_path  = "transit"
}

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_cert_file = "/home/MyUser/etc/vault-server.crt"
  tls_key_file  = "/home/MyUser/etc/vault-server.private.key"
}

api_addr = "https://vault-server-1:8200"
cluster_addr = "https://vault-server-1:8201"
ui = true
1 Like