Cannot integrate Nginx Ingress Controller with Consul (502 Bad Gateway)

Consul
1

I am trying to implement consul service mesh in my application and I am not able to make it work with Nginx Ingress. I have following config of the ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
    nginx.ingress.kubernetes.io/use-regex: "true"
  name: static-server-ingress
  namespace: default
spec:
  rules:
  - host: static.testdomain.com
    http:
      paths:
      - backend:
          service:
            name: static-server
            port:
              number: 80
        path: /*
        pathType: Prefix
  tls:
  - hosts:
    - static.testdomain.com
    secretName: wildcard-domain-tls-secret

I am using static server example from Consul portal:

apiVersion: v1
kind: Service
metadata:
  name: static-server
spec:
  selector:
    app: static-server
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: static-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: static-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: static-server
  template:
    metadata:
      name: static-server
      labels:
        app: static-server
      annotations:
        'consul.hashicorp.com/connect-inject': 'true'
        'consul.hashicorp.com/transparent-proxy': 'true'
    spec:
      containers:
        - name: static-server
          image: hashicorp/http-echo:latest
          args:
            - -text="hello world"
            - -listen=:8080
          ports:
            - containerPort: 8080
              name: http
      serviceAccountName: static-server

In Nginx I have added following annotations:

      annotations:
        consul.hashicorp.com/connect-inject: "true"
        consul.hashicorp.com/transparent-proxy: "true"
        consul.hashicorp.com/transparent-proxy-exclude-inbound-ports: 8000,80,443,9000,8443
        consul.hashicorp.com/transparent-proxy-exclude-outbound-cidrs: 10.64.0.1/32
2

I managed to find that envoy is not listening to correct port: e.g 5000 but 0.

[2021-08-04 12:47:28.656][12][debug][client] [source/common/http/codec_client.cc:43] [C11] connecting
[2021-08-04 12:47:28.656][12][debug][connection] [source/common/network/connection_impl.cc:860] [C11] connecting to 127.0.0.1:0
[2021-08-04 12:47:28.656][12][debug][connection] [source/common/network/connection_impl.cc:876] [C11] connection in progress
[2021-08-04 12:47:28.658][12][debug][connection] [source/common/network/connection_impl.cc:674] [C11] delayed connection error: 111
[2021-08-04 12:47:28.658][12][debug][connection] [source/common/network/connection_impl.cc:243] [C11] closing socket: 0

Anyone has any idea why?

3

Hello @k.morina, you also need to create a resource of type “ServiceDefaults” with “dialedDirectly” feature enabled for your service.

See what the documentation says:

If the Ingress controller acts as a LoadBalancer and routes directly to Pod IPs instead of the ClusterIP of your Kubernetes Services a ServiceDefault CRD must be applied to each backend service allowing it to use the dialedDirectly features. By default this is disabled.
Configure Ingress Controllers for Consul on Kubernetes | Consul | HashiCorp Developer

# Example Service defaults config entry
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
  name: backend
spec:
  transparentProxy:
    dialedDirectly: true

And if you check the NGINX Ingress documentation, it says this:

Why endpoints and not services¶

The NGINX ingress controller does not use Services to route traffic to the pods. Instead it uses the Endpoints API in order to bypass kube-proxy to allow NGINX features like session affinity and custom load balancing algorithms. It also removes some overhead, such as conntrack entries for iptables DNAT.
Miscellaneous - Ingress-Nginx Controller

Thus, this configuration is necessary. If I’m not mistaken, you can configure dialedDirectly: true globally, using ProxyDefaults.

I faced similar issues when deploying Kong + Consul, and solved it this way. I hope it helps.

2 Likes
4

You may still run into issues due to https://github.com/hashicorp/consul-k8s/issues/592 if the version of the nginx ingress controller you use has two kubernetes services.