Vault raft storage question

Vault
1

Hello!

I’m trying to deploy vault using DigitalOcean + nginx on a multiple droplets. I’m currently using vault file storage in one server only. I would like to migrate to raft storage and create additional servers. But i’m kinda lost because i’m trying to replicate something like in this tutorial: Vault HA cluster with integrated storage on AWS | Vault | HashiCorp Developer but i do not understand how to configure the first server. What storage should it have? I thought that i need only 3 servers and with “raft_storage” as storage in each. Each node points to another. But in this tutorial it says that i need 4 servers? Then how to configure the first one (especially the storage). Or is it safe to drop the first storage and use just 3 server with “raft_storage” in each? Also is it safe not to use SSL in vault, but use nginx with ssl and proxy?

Thank you for your help.

4

I attempted to follow this tutorial several times, however, it doesn’t work at all. Basically, you need 3 nodes, or 5, or 7, basically odd numbers above 3, since the vault doesn’t work in pairs, as it always requires more than half to be able to perform a vote and restore the fallen node. In other words, if you have 3 nodes and 1 goes down, the other 2 remain active, however, if one more goes down, it can’t stay with only 1 active. Answering your question, you can generate a TLS certificate for the Vault itself, without the need to use Nginx. All of them should contain something similar to this configuration here - #VAULT-01

ui = true

log_file = “/var/log/vault/vault.log”
log_level = “Debug”

storage “raft” {
path = “/opt/raft”
node_id = “$NAME(Is necessary different)”
retry_join {
leader_api_addr = “https://:8200$IPNODE2”
}
retry_join {
leader_api_addr = “https://:8200$IPNODE3”
}
}

api_addr = “https://:8200$IPADDRESS”
cluster_addr = “https://:8201$IPADDRESS”

listener “tcp” {
address = “0.0.0.0:8200”
tls_cert_file = “/opt/vault/tls/cert.pem”
tls_key_file = “/opt/vault/tls/key.pem”
}

log_level = “Debug”

This in the first node, in the others you will only change the parameters $IPNODE2 and $IPNODE3 referring to the other active ones.

5

Thank you for you answer!
I’m currently using self signed cert for 3 different vault instances. It works, but…
I want UI panel to be reachable only from one ip, but when i try to configure nginx i’m getting error because internal vault is using https and i’m trying to make proxy from https to https… lol. Otherwise without nginx it’s only reachable from “https://ipv4(IP of the droplet):8200” and it gives unpleasant warning of self signed non trusted cert (aka " Your connection is not private" …etc) . I was thinking about to remove tsl from all clusters and to make nginx proxy only from one server(droplet aka cluster) to the let’s encrypted ssl domain. I’m sure it will work, but it looks a bit not safe to use. Otherwise i’m stuck on how to make a proper redirect to one ‘https://vault.my-domain.com’ from the cluster1 IP.
My cluster2 config for example:

cluster_addr  = "https://{this_cluster_private_ip}:8201"

api_addr      = "https://{this_cluster_private_ip}:8200"


disable_mlock = true

 

ui = true

log_level = "Trace"

 

listener "tcp" {

  address            = "0.0.0.0:8200"

  tls_cert_file      = "/opt/vault/tls/vault-b-cert.pem"

  tls_key_file       = "/opt/vault/tls/vault-b-key.pem"

  tls_client_ca_file  = "/opt/vault/tls/vault-ca-cert.pem"

}

 

storage "raft" {

  path    = "/opt/vault/raft"

  node_id = "node2"

 

  retry_join {

    leader_tls_servername   = "vault1.my-domain.com"

    leader_api_addr         = "https://{first_cluster_private_ip}:8200"

    leader_ca_cert_file     = "/opt/vault/tls/vault-ca-cert.pem"

    leader_client_cert_file = "/opt/vault/tls/vault-b-cert.pem"

    leader_client_key_file  = "/opt/vault/tls/vault-b-key.pem"

  }

  retry_join {

    leader_tls_servername   = "vault2.my-domain.com"


    leader_api_addr         = "https://{second_cluster_private_ip}:8200"

    leader_ca_cert_file     = "/opt/vault/tls/vault-ca-cert.pem"

    leader_client_cert_file = "/opt/vault/tls/vault-b-cert.pem"

    leader_client_key_file  = "/opt/vault/tls/vault-b-key.pem"

  }

  retry_join {

    leader_tls_servername   = "vault3.my-domain.com"

    leader_api_addr         = "https://{third_cluster_private_ip}:8200"

    leader_ca_cert_file     = "/opt/vault/tls/vault-ca-cert.pem"

    leader_client_cert_file = "/opt/vault/tls/vault-b-cert.pem"

    leader_client_key_file  = "/opt/vault/tls/vault-b-key.pem"

  }

}

Also i’m thinking of replacing all three self signed certs with let’s encrypt version, but i kind of don’t need three different subdomains.

Also i’m a bit confused about “cluster_addr, api_addr” in each node’s config. In some guides it says that i should use only one public IP address of the main server for every config. In other guides it says that i should put different ip addresses for each node. I’m currently using different IP address for each node and it works.

My idea was to have three different servers. One for the main entry with nginx proxy to the ssl domain. And they should communicate with each other though the private ip.

6

The tutorial referenced in the initial post is describing the setup of:

  • One single-node Vault cluster
  • Another three-node Vault cluster using the previous cluster as a transit auto-unseal device

It sounds like you’ve both overlooked that part - the 4 nodes mentioned are not all part of one Vault cluster. The tutorial is poorly written, though - numbering all 4 nodes sequentially is unnecessarily confusing.

It’s up to you whether you adopt such a setup. I wouldn’t. It’s only meant as a “toy” setup for illustrating transit auto-unseal. It’s another way the tutorial is poorly written, focussing on demonstration, unsuitable for production configurations - though it does contain a warning box saying:

Demonstration only

The cluster created by these terraform files is solely for demonstration and should not be run in production.

Anyway … on to the next question:

You can’t just ask “is it safe”, because the answer is just going to be “maybe, it depends on other factors”.

You need to ask yourself whether you are concerned about traffic interception between nginx and Vault in your particular environment and configuration. Maybe you deem that to be over a sufficiently isolated network that it is unnecessary to encrypt - or maybe not.

Only with knowledge of your deployment environment and security goals, can you determine whether it is acceptable to terminate encryption at the loadbalancer.

You should not be setting any of these options unless you intend for clients to authenticate to Vault using client certificates. It does not sound like you are doing this.

Those guides are wrong.

cluster_addr is the address each nodes provides to the other nodes in the Vault cluster, for them to connect to that specific node. You absolutely must not point it to a load balancer - it should be the individual server addresses. Be aware that whatever you set here gets baked into the Raft configuration when you initially join the nodes to the cluster, and will not update properly if you subsequently change it (without removing and re-joining nodes to the Raft cluster), so use of DNS names rather than raw IP addresses is vital if IP addresses might change during cluster operation.

api_addr is mostly useless in modern Vault. It only really comes into play when a request arrives at a standby node, which for some reason fails to internally forward the request to the active node, or the client is for some mad reason sending a header to disable this behaviour. In such a scenario, the standby Vault node will issue a HTTP redirect to the api_addr of the node it thinks is active.

There is some further discussion of this topic at High Availability | Vault | HashiCorp Developer

1 Like
8

Thank you for your wisdom :pray:

P.s. Also configured nginx redirect. If someone will ever need information on how to redirect from nginx to ssl vault this is my nginx config:

http {
      	upstream backend {
	      least_conn;
	      server <ip1>8200;
	      server <ip2>:8200;
	      server <ip3>:8200;
	    }
        server {
          listen 80;
          server_name vault.my-company.com;
          return 301 https://$server_name$request_uri;
        }

server {
  listen 443 ssl http2;
  server_name vault.my-company.com;

  # SSL * cert
    ssl_certificate /etc/{path-to-letsencrypt}/vault.my-company.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/{path-to-letsencrypt}/vault.my-company.com/privkey.pem; # managed by Certbot

  location / {
    proxy_set_header        Host $host;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header        X-Forwarded-Proto $scheme;
    proxy_pass              https://backend;
    proxy_read_timeout      90;
  }

}