Consul binds to wrong private ip address

Consul
1

Hi *,

I try to convince consul to bind to the nebula interface:

bind_addr  = "10.21.20.54"
advertise_addr = "10.21.20.54"
client_addr = "10.21.20.54"

It even logs, that it is bound to this address:

May 16 15:06:17 postgresql-0 consul[20576]: ==> Starting Consul agent...
May 16 15:06:17 postgresql-0 consul[20576]:                Version: '1.20.6'
May 16 15:06:17 postgresql-0 consul[20576]:             Build Date: '2025-04-24 19:49:42 +0000 UTC'
May 16 15:06:17 postgresql-0 consul[20576]:                Node ID: 'fde3daa6-e87e-401d-ff35-7b74af5b3166'
May 16 15:06:17 postgresql-0 consul[20576]:              Node name: 'prod4-postgresql-0'
May 16 15:06:17 postgresql-0 consul[20576]:             Datacenter: 'de-west' (Segment: '')
May 16 15:06:17 postgresql-0 consul[20576]:                 Server: false (Bootstrap: false)
May 16 15:06:17 postgresql-0 consul[20576]:            Client Addr: [10.21.20.54] (HTTP: 8500, HTTPS: -1, gRPC: -1, gRPC-TLS: -1, DNS: 8600)
May 16 15:06:17 postgresql-0 consul[20576]:           Cluster Addr: 10.21.20.54 (LAN: 8301, WAN: 8302)

Still it does not bind to it (but to one of the other private ip addresses):

root@postgresql-0:/etc/consul# netstat -tulpen
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode      PID/Program name    
tcp        0      0 192.168.0.208:8301      0.0.0.0:*               LISTEN      1001       20374      20576/consul        
tcp        0      0 10.21.20.54:8500        0.0.0.0:*               LISTEN      1001       44003      20576/consul        
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      0          32320      9037/sshd: /usr/sbi 
tcp        0      0 10.21.20.54:8600        0.0.0.0:*               LISTEN      1001       67872      20576/consul        
tcp6       0      0 :::22                   :::*                    LISTEN      0          32322      9037/sshd: /usr/sbi 
udp        0      0 0.0.0.0:4242            0.0.0.0:*                           0          59643      19892/nebula        
udp        0      0 192.168.0.208:8301      0.0.0.0:*                           1001       20375      20576/consul        
udp        0      0 10.21.20.54:8600        0.0.0.0:*                           1001       20376      20576/consul        
udp        0      0 0.0.0.0:68              0.0.0.0:*                           0          10379      565/dhclient        
udp        0      0 127.0.0.1:323           0.0.0.0:*                           0          15048      710/chronyd         
udp6       0      0 ::1:323                 :::*                                0          15049      710/chronyd

What kind of sorcery is this?
Any help is appreciated

Kind regards,
Töns

2

BTW.: On Consul servers this is no problem:

    bind_addr      = "{{ GetInterfaceIP \"nebula1\" }}"
    advertise_addr = "{{ GetInterfaceIP \"nebula1\" }}"
    client_addr    = "{{ GetInterfaceIP \"nebula1\" }}"

Logs the bind:

May 16 14:51:09 triton-consul-0 systemd[1]: Started "HashiCorp Consul - A service mesh solution".
May 16 14:51:09 triton-consul-0 consul[1197]: ==> Starting Consul agent...
May 16 14:51:09 triton-consul-0 consul[1197]:                Version: '1.20.6'
May 16 14:51:09 triton-consul-0 consul[1197]:             Build Date: '2025-04-24 19:49:42 +0000 UTC'
May 16 14:51:09 triton-consul-0 consul[1197]:                Node ID: 'e81861f0-aa35-9f90-d03e-56fb9fa9ed54'
May 16 14:51:09 triton-consul-0 consul[1197]:              Node name: 'triton-consul-0'
May 16 14:51:09 triton-consul-0 consul[1197]:             Datacenter: 'de-west' (Segment: '<all>')
May 16 14:51:09 triton-consul-0 consul[1197]:                 Server: true (Bootstrap: false)
May 16 14:51:09 triton-consul-0 consul[1197]:            Client Addr: [10.21.20.52] (HTTP: -1, HTTPS: 8501, gRPC: 8502, gRPC-TLS: 8503, DNS: 8600)
May 16 14:51:09 triton-consul-0 consul[1197]:           Cluster Addr: 10.21.20.52 (LAN: 8301, WAN: 8302)

And is bound where it should:

root@triton-consul-0:~# netstat -tulpen |grep consul
tcp        0      0 10.21.20.52:8300        0.0.0.0:*               LISTEN      1000       22178      1197/consul         
tcp        0      0 10.21.20.52:8301        0.0.0.0:*               LISTEN      1000       21195      1197/consul         
tcp        0      0 10.21.20.52:8302        0.0.0.0:*               LISTEN      1000       22179      1197/consul         
tcp        0      0 10.21.20.52:8501        0.0.0.0:*               LISTEN      1000       22188      1197/consul         
tcp        0      0 10.21.20.52:8502        0.0.0.0:*               LISTEN      1000       22189      1197/consul         
tcp        0      0 10.21.20.52:8503        0.0.0.0:*               LISTEN      1000       22190      1197/consul         
tcp        0      0 10.21.20.52:8600        0.0.0.0:*               LISTEN      1000       22186      1197/consul         
udp        0      0 10.21.20.52:8301        0.0.0.0:*                           1000       21196      1197/consul         
udp        0      0 10.21.20.52:8302        0.0.0.0:*                           1000       22180      1197/consul         
udp        0      0 10.21.20.52:8600        0.0.0.0:*                           1000       22185      1197/consul
3

The problem, which arises from this is, that consul complains:

May 19 08:05:08 postgresql-0 consul[25320]: 2025-05-19T08:05:08.996Z [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to prod1-consul-0 but other probes failed, network may be misconfigured
May 19 08:05:09 postgresql-0 consul[25320]: 2025-05-19T08:05:09.836Z [ERROR] agent.client: RPC failed to server: method=Catalog.NodeServiceList server=10.21.20.50:8300 error="rpc error getting client: failed to get conn: dial tcp 192.168.0.208:0-10.21.20.50:8300: i/o timeout"
May 19 08:05:09 postgresql-0 consul[25320]: 2025-05-19T08:05:09.836Z [ERROR] agent.anti_entropy: failed to sync remote state: error="rpc error getting client: failed to get conn: dial tcp 192.168.0.208:0->10.21.20.50:8300: i/o timeout"
May 19 08:05:09 postgresql-0 consul[25320]: 2025-05-19T08:05:09.997Z [WARN]  agent.client.memberlist.lan: memberlist: Was able to connect to prod1-consul-0 but other probes failed, network may be misconfigured

Which is obvious, because it is trying to talk to the wrong network.