Consul ingress gateway route to https enabled application

I am trying to create an ingress gateway which can route the traffic to https enabled applications.

I am using conusl version 1.10.0 here is consul config:

global:
  name: consul
  datacenter: dc1
  gossipEncryption:
    secretName: "consul-gossip-encryption-key"
    secretKey: "key"
  tls:
    enabled: true
    enableAutoEncrypt: true
    verify: true
  acls:
    manageSystemACLs: true
  metrics:
    enabled: true
    enableGatewayMetrics: true

prometheus:
  enabled: true

ui:
  enabled: true
  service:
    type: 'ClusterIP'
  metrics:
    enabled: true
    provider: "prometheus"
    baseURL: http://prometheus-server

connectInject:
  enabled: true
  transparentProxy:
    defaultEnabled: false
 
controller:
  enabled: true

client:
  enabled: true
  grpc: true

ingressGateways:
  enabled: true
  defaults:
    replicas: 1
  gateways:
    - name: ingress-gateway
      service:
        type: LoadBalancer
        ports:
          - port: 443
            nodePort: null

# Use only one Consul server for local development
server:
  replicas: 1
  bootstrapExpect: 1
  disruptionBudget:
    enabled: true
    maxUnavailable: 0

and ingress yaml looks like this:

apiVersion: consul.hashicorp.com/v1alpha1
kind: IngressGateway
metadata:
  name: ingress-gateway
spec:
  tls:
    enabled: true
  listeners:
    - port: 443
      protocol: http
      services:
        - name: '*'

However the traffic routing works perfectly fine when https is not enabled in my application.
For local development I am using self singed certificates as .pfx file which are volume mounted to the deployment.

I have a micro-service architecture where micro-app 1 wants to connect to micro-app 2 through gRPC. My micro-apps are written in .NET Core 5 and to achieve this I’ve enabled https in micro-apps.

So how I can achieve this through consul IngressGateway resource that it starts routing incoming traffic to my https micro-app.

I always get no healthy upstream while trying to hit through localhost

I am stuck here since I am new to consul IngressGateway.

Hi @itsbibeksaini! I’m sorry this is not working for you. One thing that I noticed in your steps to reproduce that you missed was adding an intention since ACLs are enabled. Here is a guide for ingress gateways and some information on intentions, also you may have to add servicedefaults which is referenced just above this link. Let me know if you’re able to get this up and running, cheers!

Hi @kschoche1

Yes I’ve configured ServiceIntentions and ServiceDefaults as well. Sorry I didn’t posted complete configuration. So here are my rest of config:

apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceIntentions
metadata:
  name: app1
spec:
  destination:
    name: app1
  sources:
    - name: ingress-gateway
      action: allow
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
  name: app1
spec:
  protocol: http
apiVersion: v1
kind: ServiceAccount
metadata:
  name: app1
apiVersion : apps/v1
kind: Deployment
metadata:
  name: app1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app1
  template:
    metadata:
      labels:
        app: app1
      annotations:
        "consul.hashicorp.com/connect-inject": "true"
        "consul.hashicorp.com/service-name": "app1"
    spec:
      containers:
        - name: app1
          image: app1:v1
          env:
            - name: "ASPNETCORE_URLS"
              value: "http://[*]:8080"
          ports:
          - containerPort: 8080
            protocol: TCP
apiVersion: v1
kind: Service
metadata:
    name: app1
spec:
    type: ClusterIP
    ports:
    - port: 8080
      targetPort: 8080
      protocol: TCP
    selector:
        app: app1

In similar way I’ve bunch of apps in Microservices Architecture and I am using consul’s service discovery and Ingress gateway.

As you can see in diagram all micro apps in cluster are running on HTTP and while sending request I am adding Host to headers. This works perfectly fine and all traffic is being routed to concerned service.

However, if I run these micro apps on HTTPS and its deployment is as below:

apiVersion : apps/v1
kind: Deployment
metadata:
  name: app1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app1
  template:
    metadata:
      labels:
        app: app1
      annotations:
        "consul.hashicorp.com/connect-inject": "true"
        "consul.hashicorp.com/service-name": "app1"
    spec:
      containers:
        - name: app1
          image: app1:v1
          env:
            - name: "ASPNETCORE_URLS"
              value: "https://[*]:8080"
            - name: "ASPNETCORE_Kestrel__Certificates__Default__Password"
              value: "testPassword123@"
            - name: "ASPNETCORE_Kestrel__Certificates__Default__Path"
              value: "secret/appSecret.pfx"
          ports:
          - containerPort: 8080
            protocol: TCP
          volumeMounts:
          - mountPath: /app/secret
            name: app-secret
      volumes:
        - name: app-secret
          secret:
            secretName: self-signed-generic-secret

k8s secret contains pix certificate which are used by .NET Core app to run on HTTPS
so now all the micro apps use this certificate and are running on HTTPS:

so here comes when all the redirection stops and I get this message:

The reason I need to run apps on HTTPS is these micro apps make use of gRPC for internal communication and .NET Core requires TLS secured channel for that
so that’s how my desired communication looks like:

Hi @itsbibeksaini , just chipping in here as I ran into the same issue yesterday and was watching this thread but have found a solution which works for me:
If you set your grpc channel credentials to be insecure (no tls), and thus don’t provide call credentials then grpc works fine. You can then set up an interceptor on the client to add the required headers to the request metadata (for us, this is a JWT auth header). Alternatively, you can pass this metadata through with each request (I didn’t think this worked at first but turns out grpc lower cases the header keys).

One caveat with this approach is that I don’t think the web host (in our case Kestrel) can differentiate between http1.1 and http2 requests without tls, so I think you’ll need to configure your grpc port to explicitly listen for http2 requests and use a different port for any standard http requests. These should both be picked up and handled nicely by MVC so not a big issue.

I hope this potentially helps your setup.

I would still be interested in support being added for the sidecar proxy to be able to TLS to the service, potentially just by adding a ca_path to the config (or with an option to skip_verify on the cert). Just feels like it would be more flexible with what you put behind your proxy this way!

2 Likes

Hi, yeah like @saltxwater said we don’t support routing from the app’s sidecar envoy proxy to the app over TLS. This is because Connect already encrypts the traffic between the services when the request goes through each proxy.

Thanks @saltxwater I’ve tried setting up that and when I run my apps on localhost everything works fine on http2 port however when I take this configuration to Kubernetes I always get the server didn’t completed the http2 handshake

and here is code snippet where I’ve configured http2 port:
Screenshot 2021-07-10 at 12.07.16 AM

and gRPC channel:
Screenshot 2021-07-10 at 8.20.17 PM

where app1 is name of Kubernetes service.

Hi @itsbibeksaini , I was able to recreate your error about the http2 handshake, and fixed it by setting the consul connect protocol to “grpc” on the server proxy.

1 Like

Finally I made it working :partying_face:
I am posting my solution here and it might help someone :slightly_smiling_face:

While using ASP.NET, HTTP/2 gRPC won’t work with envoy proxy.
To make it work I’ve made use of gRPC-web and gRPC-web client
Screenshot 2021-07-22 at 11.42.18 PM

As given in image assume browser is your internal gRPC client which will make request using HTTP/2 but envoy will not convert HTTP request to HTTP/2 for gRPC server.
And to do this conversion from HTTP to HTTP/2 gRPC web comes to play.

For more information regarding the implementation please follow the official Microsoft document on this:

2 Likes