I’m a little unclear as to whether the HA param api_addr needs to be an IP address or a hostname?
Previously we used an IP address (api_addr = "https://10.23.49.6:9200") and everything has worked fine - specifically the consul service registration worked as intended, e.g.
# host pipeline.vault.service.consul
pipeline.vault.service.consul has address 10.22.49.11
pipeline.vault.service.consul has address 10.23.49.6
pipeline.vault.service.consul has address 10.23.49.8
However, last night I changed api_addr to use the hostname (api_addr = "https://vault-dbsa100.prod:9200") and now the consul service is returning a single record, e.g.
# host pipeline.vault.service.consul.prod
pipeline.vault.service.consul.prod is an alias for vault101.prod.
vault101.prod has address 10.23.21.105
Each call to the host command above rotates through the three different nodes in the cluster.
I feel like this is wrong now and I should have stuck with an IP address for api_addr. Is that correct?
The API address is whatever hostname (or IP address) the clients will use to connect to Vault. Vault itself never use it for communicating internally, with cluster members or plugins.
Say your Vault is behind a load-balancer. Vault does not know what is in front of it on the network. If it returned its listen address, it will most likely be impossible to resolve for external clients. You have to tell Vault how it can be reached on your network. That’s the API address.
Here is a test you can make: set the X-Vault-No-Request-Forwarding header and make a request to a follower node, Vault will 307 Redirect you to the API address.
True, but Consul is not a plugin. A plugin is a binary that Vault launches to provide some service not included with Vault, like this Enigma machine plug-in I wrote.
Back to Consul : My understanding from your question is that you use Consul to resolve the DNS request (made by host). If that’s the case, I would make the educated guess that Consul will return the active node all the time. Kill or step-down the leader and see if the IP returned by Consul follows the leader.
+Are you running Vault Opensource or Enterprise? Vault opensource will always forward to leader, so it makes sense for Consul to return that address all the time (until the leader changes).