TLS configuration for HA (with Raft)

Vault
1

I’m a bit lost for the TLS configuration. I have setup a two nodes cluster, running Vault 1.12.2 on Ubuntu 22.04. Config on one of the nodes:

storage "raft" {
  path = "/opt/vault/data/vault2"
  node_id = "vault2"
}
listener "tcp" {
  address     = "0.0.0.0:8200"
  cluster_address = "0.0.0.0:8201"
  tls_cert_file = "/etc/ssl/certs/wildcard.crt"
  tls_key_file = "/etc/ssl/private/wildcard.key"
  tls_disable_client_certs = true
}
api_addr = "https://vault2.xxxx.com:8200"
cluster_addr = "https://vault2.xxxx.com:8201"
ui = true
enable_response_header_raft_node_id = true
cluster_name = "xxxx"
disable_mlock = true
telemetry {
  prometheus_retention_time = "30s"
  disable_hostname = true
}

But I’m getting:

Jan 11 13:40:39 vault2 vault[144175]: 2023-01-11T08:40:39.539-0500 [ERROR] core: forward request error: error="error during forwarding RPC request"
Jan 11 13:49:42 vault2 vault[144175]: 2023-01-11T08:49:42.067-0500 [ERROR] core: error during forwarded RPC request: error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing x509: certificate is valid for fw-066fe9e9-b7ce-549a-fb2e-9c335a481eac, not fw-d1f1e82d-478f-15d3-8e56-1ba6749d4cf0\""

I tried by adding the nodes in retry_join blocks, but I get the same error.

Also tried this:

vault operator raft join -leader-ca-cert="$CA_CERT" https://vault1.xxxx.com:8200

Where CA_CERT is the content of the certificate in /opt/vault/tls/tls.crt from vault1, same error.

So, what is the correct configuration for Raft clustering with TLS?

2

The “fw-xxxxx”, isn’t a Proxy/FW with a SSL inspection ??

3

any solution, we are running into same issue.

4

@sarbhika.dangol You should start a new topic, and fully describe your problem and your configuration. This keeps things clear, as often users have superficially similar issues with different causes.

5

@pascalrobert

Could you let us know how you generated both server’s certificates and ideally paste the output of openssl x509 -in <cert_file>.pem -noout -text for each one.

6

A two node Raft cluster is inadvisable, as if either node fails, the entire cluster will be inoperable.

In general I would discourage people from setting the Raft node ID in the configuration file. If you leave it to Vault to manage, a random UUID will be assigned and stored in the Vault data directory. Setting it in the config file just opens up the possibility of accidentally using the same value on multiple nodes. You cannot easily migrate an existing cluster, but something to bear in mind for the future.

7

@macmiranda The error message that @pascalrobert showed mentioned request forwarding - request forwarding doesn’t use the user-supplied certificate, Vault internally generates its own.

As @Joffrey mentioned, the mystery hostnames beginning with fw- in the error message are a pretty solid hint that there is some network interception going on in this environment which is sabotaging the communication.

8

Noted.

Just curious as to why the inspection device would change the subject name in the cert.

9

I seem to have the same problem.