# Consul-helm: issue terminating TLS at AWS Load Balancer

**URL:** <https://discuss.hashicorp.com/t/consul-helm-issue-terminating-tls-at-aws-load-balancer/9749>\
**Category:** Consul\
**Tags:** k8s\
**Created:** [June 9, 2020, 11:22am UTC](https://discuss.hashicorp.com/t/consul-helm-issue-terminating-tls-at-aws-load-balancer/9749 "2020-06-09T11:22:43Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![thecosmicfrog](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/thecosmicfrog/32/7287_2.png) [@thecosmicfrog](https://discuss.hashicorp.com/u/thecosmicfrog)\
**Post date:** [June 9, 2020, 11:22am UTC](https://discuss.hashicorp.com/t/consul-helm-issue-terminating-tls-at-aws-load-balancer/9749/1 "2020-06-09T11:22:43Z")

</div>

Hi folks,

I’m having an issue with **consul-helm** and enabling TLS termination at my AWS Elastic Load Balancer for the Consul UI.

When I open the DNS for Consul UI in a browser (e.g. [https://consul.company.com](https://consul.company.com)) I get the error:

```nohighlight
Client sent an HTTP request to an HTTPS server.

```

It seems that the HTTPS-\>HTTP listener being used is always routing to the Consul NodePort running a HTTPS server. Instead, the HTTPS connection should be terminated at the load balancer, “stepped down” to HTTP and sent to the appropriate Consul HTTP port.

Here is my **values-override.yaml** file which I pass to consul-helm. Note the added **annotations** for the UI Service.

```auto
global:
  datacenter: sandbox

  gossipEncryption:
    secretName: "consul"
    secretKey: "CONSUL_GOSSIP_ENCRYPTION_KEY"

  tls:
    enabled: true
    httpsOnly: false
    enableAutoEncrypt: true
    serverAdditionalDNSSANs: ["'consul.service.consul'"]

  acls:
    manageSystemACLs: true

server:
  replicas: 3
  bootstrapExpect: 3
  storage: 20Gi

dns:
  clusterIP: 172.20.53.53

ui:
  enabled: true
  service:
    type: 'LoadBalancer'
    annotations: |
      "external-dns.alpha.kubernetes.io/hostname": "consul.company.com"
      "external-dns.alpha.kubernetes.io/ttl": "30"
      "service.beta.kubernetes.io/aws-load-balancer-backend-protocol": "http"
      "service.beta.kubernetes.io/aws-load-balancer-ssl-cert": "<CERTIFICATE_ARN>"
      "service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy": "ELBSecurityPolicy-TLS-1-2-2017-01"
      "service.beta.kubernetes.io/aws-load-balancer-ssl-ports": "https"
      "service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags": "Foo=Bar,Environment=sandbox,Name=consul-public"

syncCatalog:
  enabled: true

```

As you can see, I have the **tls.httpsOnly** key set to _false_.

When I look at my ELB config, I see a valid HTTPS-\>HTTP listener but, evidently, the **Instance Port** that it’s routing to is a HTTPS port, not HTTP, hence the browser error.

 ![Screenshot 2020-06-09 at 12.18.19](https://global.discourse-cdn.com/hashicorp/original/2X/f/f7d3219ef53e73565132c5823471cfbcae4afce0.png)

Have I misconfigured something here?

Thanks - Aaron

---

<div class="post-metadata">

**Author:** ![lkysow](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/lkysow/32/1105_2.png) [@lkysow](https://discuss.hashicorp.com/u/lkysow)\
**Post date:** [June 9, 2020, 4:52pm UTC](https://discuss.hashicorp.com/t/consul-helm-issue-terminating-tls-at-aws-load-balancer/9749/2 "2020-06-09T16:52:20Z")

</div>

Looks like we set the service’s port to 8501 when TLS is enabled in the helm chart: [https://github.com/hashicorp/consul-helm/blob/master/templates/ui-service.yaml#L23-L32](https://github.com/hashicorp/consul-helm/blob/master/templates/ui-service.yaml#L23-L32).

For now you’ll need to manually create another service with the right port. I’ll open up an issue to track: [https://github.com/hashicorp/consul-helm/issues/489](https://github.com/hashicorp/consul-helm/issues/489)

---

<div class="post-metadata">

**Author:** ![thecosmicfrog](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/thecosmicfrog/32/7287_2.png) [@thecosmicfrog](https://discuss.hashicorp.com/u/thecosmicfrog)\
**Post date:** [June 9, 2020, 6:38pm UTC](https://discuss.hashicorp.com/t/consul-helm-issue-terminating-tls-at-aws-load-balancer/9749/3 "2020-06-09T18:38:28Z")

</div>

Great, thanks @lkysow - appreciate the response.

Has there been any movement on merging any PRs which provide Kubernetes **Ingress** capabilities with Consul? This would be hugely beneficial for us.

---

<div class="post-metadata">

**Author:** ![lkysow](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/lkysow/32/1105_2.png) [@lkysow](https://discuss.hashicorp.com/u/lkysow)\
**Post date:** [June 11, 2020, 4:13pm UTC](https://discuss.hashicorp.com/t/consul-helm-issue-terminating-tls-at-aws-load-balancer/9749/4 "2020-06-11T16:13:28Z")

</div>

That’s on our backlog but unfortunately I don’t have an update as to when we’ll get to it.

---

<div class="post-metadata">

**Author:** ![ashwin-venkatesh](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/ashwin-venkatesh/32/7762_2.png) [@ashwin-venkatesh](https://discuss.hashicorp.com/u/ashwin-venkatesh)\
**Post date:** [July 23, 2020, 7:50pm UTC](https://discuss.hashicorp.com/t/consul-helm-issue-terminating-tls-at-aws-load-balancer/9749/5 "2020-07-23T19:50:24Z")

</div>

hey @thecosmicfrog,

We spent the last few days trying to get this working. We investigated switching the lb backend-protocol to be `https` but that caused the health checks to fail. If we manually updated those health check to use the `tcp` endpoint, the services became available but any request to them would just hang indefinitely.

However, we did get this working by updating the LoadBalancer service to forward both http and https requests to the port 8500 of the consul server.

```auto
apiVersion: v1
kind: Service
metadata:
  annotations:
    external-dns.alpha.kubernetes.io/hostname: aws-acm.consul.zone
    external-dns.alpha.kubernetes.io/ttl: "300"
    meta.helm.sh/release-name: consul
    meta.helm.sh/release-namespace: default
    service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: Foo=Bar,Environment=sandbox,Name=consul-public
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: <cert-arn>
    service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy: ELBSecurityPolicy-TLS-1-2-2017-01
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
  name: consul-consul-ui
  namespace: default
spec:
  clusterIP: 10.100.236.103
  externalTrafficPolicy: Cluster
  ports:
  - name: http
    nodePort: <http-port>
    port: 80
    protocol: TCP
    targetPort: 8500
  - name: https
    nodePort: <https-port>
    port: 443
    protocol: TCP
    targetPort: 8500
  selector:
    app: consul
    component: server
    release: consul
  sessionAffinity: None
  type: LoadBalancer

```

Setting the `ui.enabled` to `true` and `ui.service.enabled` to `false` should allow the use of the above service definition to route traffic to the consul server.

Having said that, this is only for the short term. We are looking into a more long term fix that would require us having a clearer idea of why the health checks fail on the ssl endpoint when the backend protocol is https and then after that, why the requests just hang indefinitely.

I hope this unblocks you but do let us know if there are follow up questions.

Cheers!!  
Ashwin

---

<div class="post-metadata">

**Author:** ![thecosmicfrog](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/thecosmicfrog/32/7287_2.png) [@thecosmicfrog](https://discuss.hashicorp.com/u/thecosmicfrog)\
**Post date:** [July 24, 2020, 1:48pm UTC](https://discuss.hashicorp.com/t/consul-helm-issue-terminating-tls-at-aws-load-balancer/9749/6 "2020-07-24T13:48:38Z")

</div>

Hey @ashwin-venkatesh. Thanks for doing some research into this. We have a user story ourselves to investigate this further and attempt mitigation, but this new information will help massively - cheers!

> We are looking into a more long term fix that would require us having a clearer idea of why the health checks fail on the ssl endpoint when the backend protocol is https and then after that, why the requests just hang indefinitely.

My halfway-educated guess for this was that the load balancer is acting as a HTTP “client” itself and attempting to open up TLS connections to the Consul server. But, as the Consul server is using a certificate not signed by a well-known CA, the load balancer closes the connection. That being said, I didn’t do a great deal of investigation into whether or not this was the case.

I’ll update here if the above solution rectifies the issue for us. At least until consul-helm supports Kubernetes Ingress (as vault-helm does already) 🙂

Thanks again! - Aaron

---

<div class="post-metadata">

**Author:** ![ashwin-venkatesh](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/ashwin-venkatesh/32/7762_2.png) [@ashwin-venkatesh](https://discuss.hashicorp.com/u/ashwin-venkatesh)\
**Post date:** [July 24, 2020, 2:05pm UTC](https://discuss.hashicorp.com/t/consul-helm-issue-terminating-tls-at-aws-load-balancer/9749/7 "2020-07-24T14:05:25Z")

</div>

@thecosmicfrog Thanks for the quick follow up.

> the load balancer is acting as a HTTP “client” itself and attempting to open up TLS connections to the Consul server. But, as the Consul server is using a certificate not signed by a well-known CA, the load balancer closes the connection.

That does sound about right 🙂

> consul-helm supports Kubernetes Ingress

~~The latest consul-helm does support ingress gateways that should potentially address this. [Here](https://www.consul.io/docs/k8s/connect/ingress-gateways) is a link to the ingress-gateway docs. It might be a little more work though as it requires further configuration on the consul server. It would be great if you could give it the old college try, but the solution mentioned above should work as well.~~

Ignore the above thing. I was wrong about ingress gateways

Cheers and HAPPY FRIDAY