Terminating Gateway CAFile

Consul
,
1

There is a note on the terminating gateway page regarding the CAFile being placed on the “system trust store of of the terminating gateway container”. I what does that mean?

I put a CAFile on /etc/pki/ca-trust/source/anchors/aws_rds.pem on all nodes in my kubernetes cluster and set the kubernetes resource as follows.

apiVersion: consul.hashicorp.com/v1alpha1
kind: TerminatingGateway
metadata:
  name: terminating-gateway
spec:
  services:
    - name: myservice
      caFile: /etc/pki/ca-trust/source/anchors/aws_rds.pem

However I get error when connecting to the service. NOTE: when I remove the caFile from the kube TerminatingGateway configuration it works fine.

2

Hi, this means if the service that you’re routing to behind the terminating gateway require a TLS connection to it. In your case if it’s working that means the service you’re connecting to does not require a TLS connection.

To be complete in this answer, if your service did have TLS then caFile is referring to a file in the pod of the terminating gateway, not on the underlying nodes. If your service’s CA was a publicly trusted CA, e.g. letsencrypt, then the terminating gateway pod already has that CA included (since it’s based on alpine) and so you could set that to /etc/ssl/cert.pem which exists in the pod already.

If it’s a private CA, then you’d need to mount that in to the pod via a volume mount which you’d set via the Helm value Helm Chart Configuration | Consul by HashiCorp, e.g.

terminatingGateways:
  enabled: true
  gateways:
  - name: terminating-gateway
    extraVolumes:
    - type: secret
      name: my-secret
      items: # optional items array
      - key: key
        path: path # secret will now mount to /consul/userconfig/my-secret/path

And then use

apiVersion: consul.hashicorp.com/v1alpha1
kind: TerminatingGateway
metadata:
  name: terminating-gateway
spec:
  services:
    - name: myservice
      caFile: /consul/userconfig/my-secret/path
3

For some clarity. I am trying to connect to an AMAZON AWS RDS Postgres instance. I’ve updated the deployment to match the configuration you demonstrated. I validated the the certificate is mounting into to the terminating gateway pods and that the Terminating Gateway resource

$ kubectl exec -it consul-terminating-gateway-84d89c7f98-429kl -- /bin/sh
...
/consul/userconfig/aws-rds-ca-key # ls
awsrdsca
/consul/userconfig/aws-rds-ca-key # exit

$ kubectl describe terminatinggateway terminating-gateway
...
Spec:
  Services:
    Ca File:  /consul/userconfig/aws-rds-ca-key/awsrdsca
    Name:   awsrdsdb
Status:
  Conditions:
    Last Transition Time:  2021-01-28T21:22:02Z
    Status:                True
    Type:                  Synced
Events:                    <none>

When I mount the same secret in my application pod and set the connection string to the following I CAN connect.

postgresql://<user>:<password>@appdb.<whatever>.rds.amazonaws.com:5432/mydb?sslrootcert=/etc/certs/ca&sslmode=verify-ca

If I modify the connection to URL to localhost:1234 (1234 being the port specified in the consul connect upstream awsrdsdb:1234) it Doesn’t work.

# This does not work
postgresql://<user>:<password>@localhost:1234/mydb?sslmode=verify-ca
# Neither does not work
postgresql://<user>:<password>@localhost:1234/mydb?sslrootcert=/etc/certs/ca&sslsslmode=verify-ca
# Nor does this
postgresql://<user>:<password>@localhost:1234/mydb

I’m still missing a piece of the puzzle here.

4

I’ve tried with a second webservice. This time a keycloak server with a TLS secured API endpoint deployed with a cert from a self managed CA. With the caFile option out of the TerminatingGateway Kube Resource it works and with it defined I get

[ root@test-busybox:/ ]$ curl -kLvvv http://localhost:2345
> GET /auth/ HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:2345
> Accept: */*
> 
* Recv failure: Connection reset by peer
curl: (56) Recv failure: Connection reset by peer
[ root@test-busybox:/ ]$ curl -kLvvv https://localhost:2345
* SSLv3, TLS handshake, Client hello (1):
* Unknown SSL protocol error in connection to localhost:2345 
curl: (35) Unknown SSL protocol error in connection to localhost:2345
5

Apologies for resurrecting an old thread, but I recently investigated this issue and found that the terminating gateway cannot currently be used to initiate TLS connections to databases MySQL and PostgreSQL. A summary of my findings can be found here https://github.com/hashicorp/consul/issues/9821.

The current workaround is to have Envoy proxy the connection to the DB as an unsecured TCP connection and rely on the application within the mesh to establish encryption with the external DB at the application layer.

6

Then does it not make documentation like this

tegw-misleading-pic
tegw-misleading-pic1024×648 58.9 KB

taken from Understand terminating gateways | Consul | HashiCorp Developer misleading?

Not happy about this!

7

@okbrown_dev, it is possible to proxy MySQL or Oracle through the terminating gateway if the external service’s protocol is set to tcp and the gateway is not configured to do any TLS origination. TLS setup should be handled by the application inside of the mesh.