Consul assigns duplicate public_listener ports to envoy proxies

Consul
1

I am configuring the Connect services in our cluster and I’m finding on a random assortment of hosts the Consul assigned public_listener ports are duplicates (multiple sidecars registering with the same agent). This looks like a race condition when registering the services.

$ curl -s http://localhost:8500/v1/agent/services | python -m json.tool | grep '"Port": 21' -A5
        "Port": 21002,
        "Proxy": {
            "DestinationServiceID": "customer-merge-service",
            "DestinationServiceName": "customer-merge-service",
            "Expose": {},
            "LocalServiceAddress": "127.0.0.1",
--
        "Port": 21004,
        "Proxy": {
            "DestinationServiceID": "endpoint-common-service",
            "DestinationServiceName": "endpoint-common-service",
            "Expose": {},
            "LocalServiceAddress": "127.0.0.1",
--
        "Port": 21002,
        "Proxy": {
            "DestinationServiceID": "file-exchange-service",
            "DestinationServiceName": "file-exchange-service",
            "Expose": {},
            "LocalServiceAddress": "127.0.0.1",
--

Notice how both the file-exchange-service and customer-merge-service have been assigned the same Port value. They both have the same Connect configuration (automatic port assignment):

      connect:
        sidecar_service: {}

The agent’s sidecar_min_port and sidecar_max_port are set to the default range (21000-21255).

Further, since Consul only checks via TCP if the connect proxy is working, Consul doesn’t see the mismatch and flags everything as healthy.

Consul v1.6.2.