Add ipv6 only server to existing ipv4/ipv6 cluster

Consul
1

Hello,

I am currently facing a problem when using the dual-stack functionality of consul.

I am running a consul cluster consisting of 5 servers (ipv4/ipv6). I need to migrate the cluster to new servers. However, the new servers have only ipv6 addresses. My plan is to extend the existing cluster with the additional 5 new servers in a first step so that I can gradually adapt my applications. In a later step i want to shutoff the old servers.

The problem I face is that I can’t get 2 addresses (ipv4 + ipv6) bound per server. I imagine the configuration something like this:

config.json:
{
“client_addr”: “x.x.x.x y:y:y:y:y:y:y:y”,
“advertise_addr_ipv6”: “y:y:y:y:y:y:y:y”,
“advertise_addr_ipv4”: “x.x.x.x”,
“datacenter”: “dc1”,
“data_dir”: “/home/consul/data”,
“start_join”: [
“p.p.p.p”
],
…
…
}

Starting consul with:
~/bin/consul_1.12 agent -node node-s3 -bind “x.x.x.x y:y:y:y:y:y:y:y” -config-file ~/cfg/config.json

Error that comes then:
==> bind_addr cannot contain multiple addresses. Use ‘addresses.{dns,http,https}’ instead.

Do I bind only one ipv4 address, then of course the server is not reachable from the new ipv6 servers. If I bind an ipv6 address, then the server is not reachable from the other servers.

Do you have any ideas how I can fix this problem or where my error is? If this procedure is not possible, how can I make the migration to the new servers without a downtime?

2

Elsewhere in your message you say there existing servers are dual stack, so why are the existing servers unreachable when configured with an ipv6 address?

3

Unfortunately, I was a little too inaccurate in the description. The dual stack currently only refers to the client interfaces (https API, … there it is no problem to enter more then one address) of consul and the presence of both ipv4 and ipv6 addresses on the network interface.

For the serf LAN (port 8301) and server rpc (port 8300) I have the problem that I get only one ipv4 or one ipv6 address bound. So I cannot add an ipv6 only server node to the existing cluster unfortunately.

4

Ah, right… try using the extra options for separately defining v4 and v6 advertise addresses on the existing servers? Consul Agent Configuration Reference | Consul by HashiCorp

I’ve never had to use that myself but it sounds like it might be intended for your use case.

5

Thanks for the hint. I have already tried that. The config snippet above is the one from the “old” servers. I suspect that the advertising at that point also refers only to the client interface. Nevertheless, when I start the server, consul expects me to bind an ip address. But it cannot bind both an ipv4 and an ipv6 address.

So it happens that the node either can’t communicate with the “new” servers or can’t communicate with the “old” servers.

6

What if you bind to 0.0.0.0 ? I think that sometimes that can be interpreted as a protocol independent wildcard, depending on OS settings, though I confess I’m not totally sure of the details

7

It is also not working. And I think this is only the “wildcard” for ipv4. I get the message:

==> Multiple private IPv4 addresses found. Please configure one with ‘bind’ and/or ‘advertise’.

Edit:

I also did some more research in the source code. It seems that in the end there can only be one IP address for the cluster communication, which is bound. Therefore I must probably look for another way.

// BindAddr is used to control the address we bind to.
// If not specified, the first private IP we find is used.
// This controls the address we use for cluster facing
// services (Gossip, Server RPC)
//
// The value can be either an ip address or a go-sockaddr
// template which resolves to a single ip address.
//
// hcl: bind_addr = string
// flag: -bind string
BindAddr *net.IPAddr