Unable to communicate via Consul-Connect service mesh

Consul
,
1

I am not able to get off the ground. I was able to deploy a server and a client as members of the service mesh, but I cannot communicate between mesh members through the mesh. My config is very basic, and I am not sure if I missed a step.

CSM helm config values:

global:
  name: consul
connectInject:
  enabled: true
controller:
  enabled: true

Essentially, I deployed this:

  • Dgraph distributed graph database:
    • Alpha statefulSet (3 pods) that listen on 8080 (HTTP) and 9080 (gRPC).
    • Zero StatefulSet (3 pods) that listen on 6080 (HTTP) and 5080 (gRPC).
    • inbound/outbound 5080 and 7080 ignored as cluster members use this to communication on gRPC.
  • Test Client
    • small flask service listening on 5000 (prints ok)
    • utilities to load_data.py (gRPC), gprcurl to test grpc, and curl to test http

The injection took place with envoy-sidecar added on the client and the server pods, but I am not able to communicate to the server pods through the mesh.

I have been able successfully deploy something similar on AKS with Istio and Linkerd without any issues in less than a day. I also tried this out on GKE with NGINX Service Mesh and was able to succeed the same day, and explored more advance features. With Consul-Connect or now Consul Service Mesh, I am unable to achieve success after 1.5 weeks.