Not able to reach to a consul mesh from Kubernetes nginx ingress

Consul
1

I receive the user requests from our applications users to my ingress via the AWS Load Balancer. However I am having difficulty in sending this request to Consul Mesh (version 0.32.0).

This is my request flow

Browser → AWS LB → nginx-ingress → service → Pod (with connect-inject)

When my service is not under Consul Mesh, I am able to reach the service.

But the moment the deployment for the service is under Consul Mesh (via connect-inject), the service is unreachable (502 error) via the nginx-ingress (though I am still able to access the service through port forwarding to my local system).

If I remove this service from Consul mesh by removing the connect-inject annotations, I am able to reach the service, but then I am not able to reach the other services which are under Consul Mesh.

Service A (Not under Consul Mesh) --> Service B (Under Consul Mesh)

Above sequence is not working either. Though I am able to communicate between services inside the mesh, but not from a Kubernetes service which is outside the mesh.

What is wrong with my configuration?

2

Hi @parag!

We’ve just published some high level guidance on this configuration : Ingress Controller Integrations | Consul by HashiCorp

If you’ve already followed this, could you post a little bit more about your configuration such as the annotations you applied to the nginx deployment as well as confirmation that the ingress controller comes online correctly as well?

Cheers

3

Hi @kschoche
I followed the documentation you referred to.

I have added the following annotations to the nginx-ingress-controller deployment

spec
  template:
    metadata:
       annotations:
         consul.hashicorp.com/connect-inject: "true"
         consul.hashicorp.com/transparent-proxy: "true"
         consul.hashicorp.com/transparent-proxy-exclude-inbound-ports: 8000,80,443,9000,8443
         consul.hashicorp.com/transparent-proxy-exclude-outbound-cidrs: 10.64.0.1/32

After applying this annotations, the nginx-ingress-controller is not coming up. It remains stuck in the init stage. I have seen the documentation talking about this problem, and I think my transparent-proxy-exclude-inbound-ports, and transparent-proxy-exclude-outbound-cidrs are correct too.

On seeing the log of the container consul-connect-inject-init, I see the error

[ERROR] Timed out waiting for service registration: error=“unable to find registered connect-proxy service”

How do I debug as to what is wrong?

4

Hello @kschoche
I deleted the consul setup and did a fresh setup from beginning. This time I was able to start the ingress controller without any problem. My requests are indeed reaching the intended pods. However path for every request is coming as ‘/’, irrespective of the path in the original request. On enabling the http logging for envoy-sidecar container, I see the path in the envoy sidecar too is ‘/’.

Here are some of the configs I am using
Consul version: 0.32.0

#ServiceDefault
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
  name: webproxy
spec:
  protocol: http

#Service
apiVersion: v1
kind: Service
metadata:
  name: webproxy
spec:
  selector:
    app: webproxy
  ports:
    - protocol: TCP
      port: 80
      targetPort: 3000

#Ingress
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: webproxy
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: webmagic.dev2.flowmagic.io
    http:
      paths:
      - backend:
          serviceName: webproxy
          servicePort: 80
        path: /(.*)

#Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: webproxy
  name: webproxy
  namespace: dev-arka
spec:
  replicas: 1
  template:
    metadata:
      annotations:
        consul.hashicorp.com/connect-inject: "true"
        consul.hashicorp.com/transparent-proxy: "true"
    spec:
      containers:
      - env:
        - name: REDIS_HOST
          value: webproxy-redis
        - name: WEBMAGIC_UI_HOST
          value: webmagicui
        - name: KEYCLOAK_URL
          value: http://static-server/iam/
        image: registry.dev2.flowmagic.io/apps/webproxy
        imagePullPolicy: Always
        name: webproxy-container
        ports:
        - containerPort: 3000
          protocol: TCP
5

If the envoy proxy is only getting / then I think there’s probably an issue with the Ingress object’s config? Does nginx.ingress.kubernetes.io/rewrite-target: / work? Looking at Rewrite - NGINX Ingress Controller they are using something a bit different.

1 Like
6

Hello @lkysow Thanks for your help. My ingress configuration was indeed a problem. After fixing it, I am able to make it work as expected.

Thanks
Parag

1 Like