I have 3 consul servers on DigitalOcean with the following config:
datacenter = "dc1"
data_dir = "/opt/consul"
bind_addr = "0.0.0.0"
advertise_addr = "10.114.0.x" --> x = 1, 2, 3 respectively
client_addr = "0.0.0.0"
retry_join = ["10.114.0.1", "10.114.0.2", "10.114.0.3"]
connect {
enabled = true
}
ui = true
server = true
bootstrap_expect = 3
All 3 servers have 3 IPs: Private, Public, Floating
This is fine and I have a few clients that connect to it in the same VPC with configuration like this:
datacenter = "dc1"
data_dir = "/opt/consul"
advertise_addr = "10.114.0.4"
retry_join = ["10.114.0.1", "10.114.0.2", "10.114.0.3"]
Now, a new client comes and he has a different account/team with its own separate billing settings. This means that he has his own VPC for his account.
If I create a client for him with the client configuration above, it fails with the following:
[ERROR] agent.client: RPC failed to server: method=Catalog.ListDatacenters server=10.114.0.3:8300 error="rpc erro r getting client: failed to get conn: dial tcp <nil>->10.114.0.1:8300: connect: no route to host"
[ERROR] client: error discovering nomad servers: error="client.consul: unable to query Consul datacenters: Unexpected response code: 500 (rpc error getting client: failed to get conn: dial tcp <nil>->10.114.0.1:8300: connect: no route to host)"
It also logs this for the other 2 servers. This is not a big surprise, since they’re on a different VPC.
I tried many sensible permutations of the following:
- Use public IPs for
retry_join - Use floating IPs for
retry_join - Use public IP for
bind_addr - Use public IP for
advertise_addr
By sensible, I mean setting the 2 wan attributes together, etc.
One approach could be to create another datacenter for the client account and join the 2 with consul join -wan but that’s not what I want to do.
Is there an option to join a client outside of the VPC to a consul mesh without creating another datacenter?