Inquiries about api_addr, cluster_addr of the vault cluster

Vault
1

Hi!

I have a question during the vault cluster test, so I’ll ask.

Configuration information: 2 vaults behind LB, 3 consuls as vault storage.

Curious about the config in vault.hcl.

  1. For api_addr of vault#1 and vault#2, is it correct to write LB vip?

  2. In cluster_addr, is it correct to write the private IP of each vault#1 and vault#2?

  3. I’m not using tls. By the way, whether you write http or https in cluster_addr, status (#vault status) is all displayed as https.
    Is this the right move?

I hope you can help me!
Thanks.

2

Hello,

EDIT: Ignore this first section, as pointed out by @stuart-c below.

First, let’s talk about:

This is not a good configuration. Since Vault in HA mode elects an active node by a quorum of available nodes communicating with each other, a 2 node cluster has no more redundancy tolerance than a 1 node cluster - you’re deriving no benefit at all from the 2nd node.

Vault clusters should usually be 1 node, or 3 nodes. (If using Vault Enterprise in a scenario with high read throughput, a cluster of more than 3 nodes may make sense due to the Enterprise-only “Performance Standby” feature.)

OK, with that out of the way, let’s talk about api_addr.

In general, it’s supposed to an an address at which external clients can reach the cluster.

The main example of this is that, whatever you write in api_addr, is what Vault will issue an HTTP redirect to, if a request is routed to a standby node, and the standby node is unable to internally forward the request to the active node. However this very rarely happens - I’ve had a Vault cluster that was operational for years before we noticed the api_addr was incorrect for this use case.

This is addressed in the Vault documentation at High Availability | Vault | HashiCorp Developer, but it is not well linked, so can be hard to find if you don’t already know where to look for it.

But… Vault uses api_addr in (IMO) multiple conflicting ways. (My bug report on this: `api_addr` is used for multiple purposes with sometimes differing requirements · Issue #15070 · hashicorp/vault · GitHub)

If you have any Vault plugins configured, each plugin also needs to be able to connect to the api_addr as it starts up. IMO this is a bug as it means the local startup of processes on a node, depend on the LB, if you set api_addr to the LB.

And now, lastly, cluster_addr.

This is only used for internal communication between Vault nodes. It needs to specifically identify the individual node. A private IP is fine - a DNS name resolving to a private IP is also fine.

It is mentioned in the docs at Server Configuration | Vault | HashiCorp Developer that Vault will ignore whatever URL scheme you write in the config and override it with https://. The certificates used with this endpoint are fully managed internally to Vault, and are not the ones you may or may not configure for api_addr.

Often you don’t even need to set it at all, and Vault will autodetect something sensible.

There is an exception that, when using Raft storage (but you aren’t, you’re using Consul), Vault insists you set it manually - possibly because whatever you have set at the moment of joining the Raft cluster, gets baked into the Raft config, and doesn’t update with future changes.

3

I don’t believe that’s true.

For raft it does use a quorum based clustering setup, but for simple HA (not using raft) it just has a simple locking based leadership selection (based on one of the nodes locking something in the storage system first) with other nodes being standby nodes. Therefore there is no need to have odd numbers of nodes.

You do need 3/5 nodes if you are using the integrated raft storage.

1 Like
4

Ah, you’re right, I’d failed to appreciate the full selection of different HA behaviours Vault can use.

5

Thanks everyone for the replies!

Is my understanding correct regarding api_addr?

  • lb vip is written in api_addr, but it may be affected if the vault plugin is used.

So, what happens if you are behind a load balancer and use a plugin…?

If you are not using a plugin, is it correct to write a load balancer ip…?

6

Please have a read through this link in particular from my earlier reply: https://www.vaultproject.io/docs/concepts/ha#client-redirection

I’m happy to further discuss things that still need clarification, but it is a fairly thorough explanation of

(to which the answer is not a simple “yes” or “no”)

7

This is the cluster name, you can use the LB http://… address

You don’t need to set this unless you’re setting up an Enterprise license with PerfRepl or DR clusters.

It’s not best practice, but if you’re doing it in your private lab with no external access it’s up to you. If you or the clients accessing Vault are not on a private network then you’re sort of losing all of the gains you’re getting by adding Vault. You can just put the secrets into an HTML document and let people copy it from there.

The second part of the question, no if the status says https then vault thinks you’re using SSL. You missed tls_disable on one of your listeners.

[edit: fixed api_addr and cluster_addr ]

8

Sorry, but no, it’s not accurate to call api_addr the “cluster name”, as it is a per-node setting. You can and should set it to the LB in some deployment scenarios but often it will be different across each cluster node.

Also no, this cluster_addr is not only relevant to Enterprise replication. It’s even mandatory to set it when running an OSS Raft cluster.

And, no, tls_disable does not affect the clustering port, which always uses TLS.