Consul connect proxy gives "(52) Empty reply from server" when calling the endpoint

Consul
1

I am exploring consul for service registry and discovery to start with.
I have run a consul server on my machine inside a docker container.
On the other hand, I have a service which is dockerized and is being run through docker-compose in local.
I wanted to to register this service and enable it for service discovery. So attached one more container for consul agent and sidecar proxy in the docker-compose of the service.

I bring everything up and check the consul dashboard everything looks good.

image
image1644×914 58.7 KB

Response of http://localhost:8500/v1/catalog/services

{
    "consul": [],
    "demo-app": [],
    "demo-app-sidecar-proxy": []
}

Response of http://localhost:8500/v1/catalog/service/demo-app

[
    {
        "ID": "bb482d93-f675-0c35-802b-db60fea96f82",
        "Node": "demo-app",
        "Address": "172.25.0.4",
        "Datacenter": "dc1",
        "TaggedAddresses": {
            "lan": "172.25.0.4",
            "lan_ipv4": "172.25.0.4",
            "wan": "172.25.0.4",
            "wan_ipv4": "172.25.0.4"
        },
        "NodeMeta": {
            "consul-network-segment": "",
            "consul-version": "1.19.2"
        },
        "ServiceKind": "",
        "ServiceID": "demo-app",
        "ServiceName": "demo-app",
        "ServiceTags": [],
        "ServiceAddress": "",
        "ServiceWeights": {
            "Passing": 1,
            "Warning": 1
        },
        "ServiceMeta": {},
        "ServicePort": 4043,
        "ServiceSocketPath": "",
        "ServiceEnableTagOverride": false,
        "ServiceProxy": {
            "Mode": "",
            "MeshGateway": {},
            "Expose": {}
        },
        "ServiceConnect": {},
        "ServiceLocality": null,
        "CreateIndex": 25,
        "ModifyIndex": 25
    }
]

Service definition JSON used for the above

{
  "service": {
    "name": "demo-app",
    "port": 4043,
    "checks": [
      {
        "id": "demo-app",
        "http": "http://demo-app:4043/health",
        "interval": "5s",
        "timeout": "2s"
      }
    ]
  }
}

And http://localhost:8500/v1/catalog/service/demo-app-sidecar-proxy

[
    {
        "ID": "bb482d93-f675-0c35-802b-db60fea96f82",
        "Node": "demo-app",
        "Address": "172.25.0.4",
        "Datacenter": "dc1",
        "TaggedAddresses": {
            "lan": "172.25.0.4",
            "lan_ipv4": "172.25.0.4",
            "wan": "172.25.0.4",
            "wan_ipv4": "172.25.0.4"
        },
        "NodeMeta": {
            "consul-network-segment": "",
            "consul-version": "1.19.2"
        },
        "ServiceKind": "connect-proxy",
        "ServiceID": "demo-app-sidecar-proxy",
        "ServiceName": "demo-app-sidecar-proxy",
        "ServiceTags": [],
        "ServiceAddress": "",
        "ServiceTaggedAddresses": {
            "consul-virtual": {
                "Address": "240.0.0.1",
                "Port": 4043
            }
        },
        "ServiceWeights": {
            "Passing": 1,
            "Warning": 1
        },
        "ServiceMeta": {},
        "ServicePort": 4043,
        "ServiceSocketPath": "",
        "ServiceEnableTagOverride": false,
        "ServiceProxy": {
            "DestinationServiceName": "demo-app",
            "DestinationServiceID": "demo-app",
            "LocalServiceAddress": "demo-app",
            "LocalServicePort": 4043,
            "Mode": "",
            "MeshGateway": {},
            "Expose": {}
        },
        "ServiceConnect": {},
        "ServiceLocality": null,
        "CreateIndex": 24,
        "ModifyIndex": 24
    }
]

Proxy service definition JSON used for the above

{
  "service": {
    "name": "demo-app-sidecar-proxy",
    "port": 4043,
    "kind": "connect-proxy",
    "checks": [
      {
        "name": "Connect Sidecar Listening",
        "tcp": "127.0.0.1:4043",
        "interval": "5s",
        "timeout": "2s"
      },
      {
        "name": "Connect Sidecar Aliasing demo-app",
        "alias_service": "demo-app"
      }
    ],
    "proxy": {
      "destination_service_name": "demo-app",
      "destination_service_id": "demo-app",
      "local_service_address": "demo-app",
      "local_service_port": 4043
    }
  }
}

Now when I actually test curl http://127.0.0.1:4043/health or curl -X GET http://demo-app.service.consul:4043/health inside sidecar proxy container, then I get (52) Empty reply from server.

What am I doing wrong?

Below are the logs of consul connect proxy

consul-agent-1  | ==> Consul Connect proxy starting...
consul-agent-1  |     Configuration mode: Agent API
consul-agent-1  |         Sidecar for ID: demo-app
consul-agent-1  |               Proxy ID: demo-app-sidecar-proxy
consul-agent-1  | 
consul-agent-1  | ==> Log data will now stream in as it occurs:
consul-agent-1  | 
consul-agent-1  |     2024-09-03T14:13:04.081Z [DEBUG] proxy: got new config
consul-agent-1  |     2024-09-03T14:13:04.082Z [INFO]  proxy: Proxy loaded config and ready to serve
consul-agent-1  |     2024-09-03T14:13:04.083Z [INFO]  proxy: Parsed TLS identity: uri=spiffe://4712c92c-20a9-f067-0a7a-c318021a8aef.consul/ns/default/dc/dc1/svc/demo-app
consul-agent-1  |     2024-09-03T14:13:04.083Z [INFO]  proxy: Starting listener: listener="public listener" bind_addr=0.0.0.0:4043

I am using hashicorp/consul:1.19 docker image

2

Hi, were you able to resolve this issue. I am getting the same error.

3

No @harsh.lif3 , I did not get a solution. I tried a similar setup using minikube (k8s), it worked there.

4

Hi @abdul.ateek,

From the output it looks like you are using Consul’s built-in proxy. Using this is not recommended for production use cases. You should be using envoy proxy instead.

ref: Built-in Proxy Configuration | Service Mesh | Consul | HashiCorp Developer