I am trying to create an ingress gateway which can route the traffic to https enabled applications.
I am using conusl version 1.10.0 here is consul config:
global:
name: consul
datacenter: dc1
gossipEncryption:
secretName: "consul-gossip-encryption-key"
secretKey: "key"
tls:
enabled: true
enableAutoEncrypt: true
verify: true
acls:
manageSystemACLs: true
metrics:
enabled: true
enableGatewayMetrics: true
prometheus:
enabled: true
ui:
enabled: true
service:
type: 'ClusterIP'
metrics:
enabled: true
provider: "prometheus"
baseURL: http://prometheus-server
connectInject:
enabled: true
transparentProxy:
defaultEnabled: false
controller:
enabled: true
client:
enabled: true
grpc: true
ingressGateways:
enabled: true
defaults:
replicas: 1
gateways:
- name: ingress-gateway
service:
type: LoadBalancer
ports:
- port: 443
nodePort: null
# Use only one Consul server for local development
server:
replicas: 1
bootstrapExpect: 1
disruptionBudget:
enabled: true
maxUnavailable: 0
and ingress yaml looks like this:
apiVersion: consul.hashicorp.com/v1alpha1
kind: IngressGateway
metadata:
name: ingress-gateway
spec:
tls:
enabled: true
listeners:
- port: 443
protocol: http
services:
- name: '*'
However the traffic routing works perfectly fine when https is not enabled in my application.
For local development I am using self singed certificates as .pfx file which are volume mounted to the deployment.
I have a micro-service architecture where micro-app 1 wants to connect to micro-app 2 through gRPC. My micro-apps are written in .NET Core 5 and to achieve this I’ve enabled https in micro-apps.
So how I can achieve this through consul IngressGateway resource that it starts routing incoming traffic to my https micro-app.
I always get no healthy upstream while trying to hit through localhost
I am stuck here since I am new to consul IngressGateway.
Hi @itsbibeksaini! I’m sorry this is not working for you. One thing that I noticed in your steps to reproduce that you missed was adding an intention since ACLs are enabled. Here is a guide for ingress gateways and some information on intentions, also you may have to add servicedefaults which is referenced just above this link. Let me know if you’re able to get this up and running, cheers!
Hi @kschoche
Yes I’ve configured ServiceIntentions and ServiceDefaults as well. Sorry I didn’t posted complete configuration. So here are my rest of config:
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceIntentions
metadata:
name: app1
spec:
destination:
name: app1
sources:
- name: ingress-gateway
action: allow
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
name: app1
spec:
protocol: http
apiVersion: v1
kind: ServiceAccount
metadata:
name: app1
apiVersion : apps/v1
kind: Deployment
metadata:
name: app1
spec:
replicas: 1
selector:
matchLabels:
app: app1
template:
metadata:
labels:
app: app1
annotations:
"consul.hashicorp.com/connect-inject": "true"
"consul.hashicorp.com/service-name": "app1"
spec:
containers:
- name: app1
image: app1:v1
env:
- name: "ASPNETCORE_URLS"
value: "http://[*]:8080"
ports:
- containerPort: 8080
protocol: TCP
apiVersion: v1
kind: Service
metadata:
name: app1
spec:
type: ClusterIP
ports:
- port: 8080
targetPort: 8080
protocol: TCP
selector:
app: app1
In similar way I’ve bunch of apps in Microservices Architecture and I am using consul’s service discovery and Ingress gateway.
As you can see in diagram all micro apps in cluster are running on HTTP and while sending request I am adding Host to headers. This works perfectly fine and all traffic is being routed to concerned service.
However, if I run these micro apps on HTTPS and its deployment is as below:
apiVersion : apps/v1
kind: Deployment
metadata:
name: app1
spec:
replicas: 1
selector:
matchLabels:
app: app1
template:
metadata:
labels:
app: app1
annotations:
"consul.hashicorp.com/connect-inject": "true"
"consul.hashicorp.com/service-name": "app1"
spec:
containers:
- name: app1
image: app1:v1
env:
- name: "ASPNETCORE_URLS"
value: "https://[*]:8080"
- name: "ASPNETCORE_Kestrel__Certificates__Default__Password"
value: "testPassword123@"
- name: "ASPNETCORE_Kestrel__Certificates__Default__Path"
value: "secret/appSecret.pfx"
ports:
- containerPort: 8080
protocol: TCP
volumeMounts:
- mountPath: /app/secret
name: app-secret
volumes:
- name: app-secret
secret:
secretName: self-signed-generic-secret
k8s secret contains pix certificate which are used by .NET Core app to run on HTTPS
so now all the micro apps use this certificate and are running on HTTPS:
so here comes when all the redirection stops and I get this message:
The reason I need to run apps on HTTPS is these micro apps make use of gRPC for internal communication and .NET Core requires TLS secured channel for that
so that’s how my desired communication looks like:
Hi @itsbibeksaini , just chipping in here as I ran into the same issue yesterday and was watching this thread but have found a solution which works for me:
If you set your grpc channel credentials to be insecure (no tls), and thus don’t provide call credentials then grpc works fine. You can then set up an interceptor on the client to add the required headers to the request metadata (for us, this is a JWT auth header). Alternatively, you can pass this metadata through with each request (I didn’t think this worked at first but turns out grpc lower cases the header keys).
One caveat with this approach is that I don’t think the web host (in our case Kestrel) can differentiate between http1.1 and http2 requests without tls, so I think you’ll need to configure your grpc port to explicitly listen for http2 requests and use a different port for any standard http requests. These should both be picked up and handled nicely by MVC so not a big issue.
I hope this potentially helps your setup.
I would still be interested in support being added for the sidecar proxy to be able to TLS to the service, potentially just by adding a ca_path to the config (or with an option to skip_verify on the cert). Just feels like it would be more flexible with what you put behind your proxy this way!
2 Likes
Hi, yeah like @saltxwater said we don’t support routing from the app’s sidecar envoy proxy to the app over TLS. This is because Connect already encrypts the traffic between the services when the request goes through each proxy.
Thanks @saltxwater I’ve tried setting up that and when I run my apps on localhost everything works fine on http2 port however when I take this configuration to Kubernetes I always get the server didn’t completed the http2 handshake
and here is code snippet where I’ve configured http2 port:
and gRPC channel:
where app1 is name of Kubernetes service.
Hi @itsbibeksaini , I was able to recreate your error about the http2 handshake, and fixed it by setting the consul connect protocol to “grpc” on the server proxy.
1 Like
Finally I made it working 
I am posting my solution here and it might help someone 
While using ASP.NET, HTTP/2 gRPC won’t work with envoy proxy.
To make it work I’ve made use of gRPC-web and gRPC-web client
As given in image assume browser is your internal gRPC client which will make request using HTTP/2 but envoy will not convert HTTP request to HTTP/2 for gRPC server.
And to do this conversion from HTTP to HTTP/2 gRPC web comes to play.
For more information regarding the implementation please follow the official Microsoft document on this:
2 Likes
