Not able to communicate between microservices hosted in minikube single cluster

Consul
1

I have two apis (api1 and api2) configured and deployed in minikube cluster. Api1 wants to communicate to Api2. I have already configured the annotations and intentions too. However, nothing worked.
my consul config file configuration

global:
  name: consul
  datacenter: testcenter
  image: hashicorp/consul:1.10.3
  imageEnvoy: envoyproxy/envoy:v1.20.0
  metrics:
    enabled: true
    enableAgentMetrics: true
server:
  replicas: 1
ui:
  enabled: true
connectInject:
  enabled: true
  default: false
syncCatalog:
  enabled: false
controller:
  enabled: true
prometheus:
  enabled: false
grafana:
  enabled: false

api-1

apiVersion: v1
kind: Service
metadata:
  name: order-service
spec:
  selector:
    app: order-service
  ports:
    - protocol: TCP
      port: 5000
      targetPort: 80
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: order-service
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
  labels:
    app: order-service
spec:
  replicas: 1
  selector:
    matchLabels:
      app: order-service
  template:
    metadata:
      labels:
        app: order-service
      annotations:
        'consul.hashicorp.com/connect-inject': 'true'        
        'consul.hashicorp.com/connect-service-upstreams': 'payment-service:5001'      
    spec:
      containers:
      - name: orderapi
        image: image/orderapi:1.4
        env:
        - name: PaymentBaseUrl
          value: "http://payment-service:5001/"
        ports:
        - containerPort: 5000

api 2 config

apiVersion: v1
kind: Service
metadata:
  name: payment-service
spec:
  selector:
    app: payment-service
  ports:
    - protocol: TCP
      port: 5001
      targetPort: 80      
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: payment-service
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
  labels:
    app: payment-service
spec:
  replicas: 1
  selector:
    matchLabels:
      app: payment-service
  template:
    metadata:
      labels:
        app: payment-service
      annotations:
        'consul.hashicorp.com/connect-inject': 'true'               
    spec:
      containers:
      - name: paymentapi
        image: image/paymentapi:1.4
        ports:
        - containerPort: 5001
2

Hello!

First off, welcome to the consul community! Sorry about the issue you’re having, could you provide us with the error message/logs you’re seeing when running the yaml files above?

3

Hey Amier,
It works perfectly (api-1 can communicate to api-2) when I remove the “connect-inject” annotations. However, It’s not working with consul injections.
I can see it in the logs, prematurely closed the connection whilst api-1 connecting to api-2 but I get responses from both apis, if I trigger those separately from postman

4

@krishonline One thing that stands out is the Envoy version you are using. Consul 1.10 dosen’t support Envoy 1.20. You can see a list of the supported versions here

5

Hey @karl-cardenas-coding
thanks. I tried, downgraded to 1.18.4 but still same.

6

@krishonline I wonder if the application is pointing to localhost ?

I see that the env variable PaymentBaseUrl is pointing to the payment-service. Try changing that to localhost http://localhost:5001. That way the traffic will go through the sidecar. Make sure you add the consul annotations back though.

        - name: PaymentBaseUrl
          value: "http://payment-service:5001/"
7

@karl-cardenas-coding
yeah, tried but still same error “prematurely closed”.
Do I need to enable anything else in the consul config file?

8

@krishonline could you share the Envoy output logs please.

9

envoy_logs.txt (16.5 KB)

@karl-cardenas-coding envoy logs attached

10

So according to the logs Envoy is coming up fine, at least for the payment service. I’d be curious to see what order-service logs are saying.

[2021-11-16 03:38:44.949][1][info][upstream] [source/server/lds_api.cc:78] lds: add/update listener 'payment-service:127.0.0.1:5001'
[2021-11-16 03:38:44.950][1][info][config] [source/server/listener_manager_impl.cc:888] all dependencies initialized. starting workers

So if you changed the environment variable to PaymentBaseUrl to http://localhost:5001 , can you see if the order-service application is returning any errors in the logs that could provide clues? Same question for the order-service?

11

order-service-logs.txt (3.8 KB)
payment-envoy.txt (16.9 KB)

@karl-cardenas-coding
attached order service logs and payment envoy logs

12

@krishonline Envoy started up fine, per the log file. However, in your order-service logs I see it’s returning an error

←[30mfail←[39m←[22m←[49m: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "9HP36TERD", Request id "9HP36TERD:00000003": An unhandled exception was thrown by the application.
      System.Net.Http.HttpRequestException: An error occurred while sending the request.

Let’s try this. Can you set the payment service ports to be equal.

 selector:
    app: payment-service
  ports:
    - protocol: TCP
      port: 5001
      targetPort: 5001

I am thinking that might be problem, as the service is expecting traffic on port 80 in the existing definition. But everything we are providing the order-service is pointing to 5001. So either make them the same, or change the order-service environment variable to http://payment-service:80/ But I would recommend sticking to 5001 as port 80 is a lower port and usually requires elevated permissions to open up.

13

@karl-cardenas-coding Thanks a heap. Exposing the ports (5000 and 5001) explicitly did the trick.