Not able to reach to a consul mesh from Kubernetes nginx ingress

I receive the user requests from our applications users to my ingress via the AWS Load Balancer. However I am having difficulty in sending this request to Consul Mesh (version 0.32.0).

This is my request flow

Browser → AWS LB → nginx-ingress → service → Pod (with connect-inject)

When my service is not under Consul Mesh, I am able to reach the service.

But the moment the deployment for the service is under Consul Mesh (via connect-inject), the service is unreachable (502 error) via the nginx-ingress (though I am still able to access the service through port forwarding to my local system).

If I remove this service from Consul mesh by removing the connect-inject annotations, I am able to reach the service, but then I am not able to reach the other services which are under Consul Mesh.

Service A (Not under Consul Mesh) --> Service B (Under Consul Mesh) 

Above sequence is not working either. Though I am able to communicate between services inside the mesh, but not from a Kubernetes service which is outside the mesh.

What is wrong with my configuration?

Hi @parag!

We’ve just published some high level guidance on this configuration : Ingress Controller Integrations | Consul by HashiCorp

If you’ve already followed this, could you post a little bit more about your configuration such as the annotations you applied to the nginx deployment as well as confirmation that the ingress controller comes online correctly as well?


Hi @kschoche
I followed the documentation you referred to.

I have added the following annotations to the nginx-ingress-controller deployment

       annotations: "true" "true" 8000,80,443,9000,8443

After applying this annotations, the nginx-ingress-controller is not coming up. It remains stuck in the init stage. I have seen the documentation talking about this problem, and I think my transparent-proxy-exclude-inbound-ports, and transparent-proxy-exclude-outbound-cidrs are correct too.

On seeing the log of the container consul-connect-inject-init, I see the error

[ERROR] Timed out waiting for service registration: error=“unable to find registered connect-proxy service”

How do I debug as to what is wrong?

Hello @kschoche
I deleted the consul setup and did a fresh setup from beginning. This time I was able to start the ingress controller without any problem. My requests are indeed reaching the intended pods. However path for every request is coming as ‘/’, irrespective of the path in the original request. On enabling the http logging for envoy-sidecar container, I see the path in the envoy sidecar too is ‘/’.

Here are some of the configs I am using
Consul version: 0.32.0

kind: ServiceDefaults
  name: webproxy
  protocol: http
apiVersion: v1
kind: Service
  name: webproxy
    app: webproxy
    - protocol: TCP
      port: 80
      targetPort: 3000
apiVersion: extensions/v1beta1
kind: Ingress
  name: webproxy
  annotations: /
  - host:
      - backend:
          serviceName: webproxy
          servicePort: 80
        path: /(.*)
apiVersion: apps/v1
kind: Deployment
    app: webproxy
  name: webproxy
  namespace: dev-arka
  replicas: 1
      annotations: "true" "true"
      - env:
        - name: REDIS_HOST
          value: webproxy-redis
        - name: WEBMAGIC_UI_HOST
          value: webmagicui
        - name: KEYCLOAK_URL
          value: http://static-server/iam/
        imagePullPolicy: Always
        name: webproxy-container
        - containerPort: 3000
          protocol: TCP

If the envoy proxy is only getting / then I think there’s probably an issue with the Ingress object’s config? Does / work? Looking at Rewrite - NGINX Ingress Controller they are using something a bit different.

1 Like

Hello @lkysow Thanks for your help. My ingress configuration was indeed a problem. After fixing it, I am able to make it work as expected.


1 Like