Failing sidecar listening check with envoy and IPv6 bind_address

Consul
1

I have been rolling out Consul in an IPv6 single-stack datacenter. For the most part I have all the services we wanted to evaluate working, but I have hit a blocker with service mesh and was hoping I could get some feedback here on if this is a bug I should write up and if there is a work-around.

It looks like the “sidecar listening” check is getting a blank IP address to connect to when envoy is configured to listen on IPv6.

The service mesh stuff is very compelling! I have considered getting IPv4 addressing rolled out to test it, but in this environment that has no IPv4 routing, firewall rules, address assignments, etc, it would be a big undertaking. So many thanks in advance for any input on getting this to work in an IPv6 environment!

Service configuration (noweb socket is a web server listening on a unix socket):

(tokens have been replaced by random uuids below)

service {
    name = "cprox"
    socket_path = "/var/run/noweb/noweb.sock"

    connect {
        sidecar_service {
            proxy {
                config {
                    bind_address = "::"
                }
            }
        }
    }

    check {
        name = "cprox-curl"
        args = ["/usr/bin/curl", "--unix-socket", "/var/run/noweb/noweb.sock", "http://a/"]
        interval = "10s"
        timeout = "1s"
    }

    token = "c29582b8-a631-11ed-b68a-cfaaf07563d5"
}

Starting the sidecar:

consul connect envoy -sidecar-for=cprox -grpc-addr=unix:///var/run/consul/consul_grpc.sock

Envoy listens on [::]:21001

[2023-02-06 08:03:21.764][1528043][info][upstream] [source/server/lds_api.cc:82] lds: add/update listener 'public_listener::::21001'

The Consul check continues to fail and appears to be trying to connect to a blank IP address:

Feb  6 08:08:08 gurp2 consul[1448512]: 2023-02-06T08:08:08.262-0600 [WARN]  agent: Check socket connection failed: check=service:cprox-sidecar-proxy:1 error="dial tcp :21001: connect: connection refused"

Things I have tried:

  • bind_address = "[::]" - Envoy does not accept that formatting
  • an explicit IP address - Envoy listens on that IP explicitly, but the consul check still uses a blank address
  • go-sockaddr template - Does not get parsed by Consul for the bind_address property and it is passed literally to envoy, which rejects it
2

Workaround for now is to override the default registered checks:

service {
  name = "backend"
  port = 7000

  connect {
    sidecar_service {
      proxy { config { bind_address = "::" } }
      port   = 22000
      checks = {
        Interval = "10s"
        Name     = "Connect Sidecar Listening"
        TCP      = "[::1]:22000"
      }
      checks = {
        alias_service = "backend"
        name          = "Connect Sidecar Aliasing backend"
      }
    }
  }
}
3

Also hardcoding the address to the ipv6 address works:

service {
  name = "backend"
  port = 7000
  address = "2600:1f16:e6d:df00:eaa7:1563:14ae:641d"
  connect {
    sidecar_service {
      port = 22000
    }
  }
}