Consul verified TLS from Pods in Kubernetes cluster

Consul
1

Hi all,

I’m having some difficulty understanding Consul end-to-end TLS. For reference, I’m using Consul in Kubernetes (via the hashicorp/consul Helm chart). Only one datacenter and Kubernetes cluster - no external parties or concerns.

I have configured my override values.yaml file like so:

global:
  datacenter: sandbox

  gossipEncryption:
    secretName: "consul"
    secretKey: "CONSUL_GOSSIP_ENCRYPTION_KEY"

  tls:
    enabled: true
    httpsOnly: true
    enableAutoEncrypt: true
    serverAdditionalDNSSANs: ["'consul.service.consul'"]

server:
  replicas: 3
  bootstrapExpect: 3
  storage: 20Gi

dns:
  clusterIP: 172.20.53.53

ui:
  service:
    type: 'LoadBalancer'

syncCatalog:
  enabled: true

All other values are as default from the shipped values.yaml file.

This works, and Consul client logs suggest that all agents area connecting nicely using TLS, with relevant certs and keys being created by (as I understand) the Auto-encryption feature of Consul.

What I don’t understand is how to initiate a HTTPS connection from an application on Kubernetes, running in a Pod, to a Consul server. Since the Pod’s container does not (presumably) have the Consul root CA cert in its trust store, all HTTPS calls fail, as per wget example below:

# Connect to Pod:
laptop$> kubectl exec -it my-pod sh

# Attempt valid HTTPS connection:
my-pod$> wget -q -O - https://consul.service.consul:8501
Connecting to consul.service.consul:8501 (10.110.1.131:8501)
ssl_client: consul.service.consul: certificate verification failed: unable to get local issuer certificate
wget: error getting response: Connection reset by peer

# Retry, but ignore certificate validity issues:
my-pod$> wget --no-check-certificate -q -O - https://consul.service.consul:8501/v1/status/leader
"10.110.1.131:8300"

How am I supposed to enforce end-to-end (verified) HTTPS connections from my apps on Kubernetes to Consul if the container does not recognize the certificate as valid?
Am I misunderstanding something about certificate propagation?

Many thanks.

2

Hi @thecosmicfrog I’m David from the Consul Kubernetes PM team here. Could you help us understand what use case were looking to address for communicating with the Consul Server API? We were under the assumption that auto-encrypt would help with mTLS traffic between the services and that that Consul on Kubernetes would take care of all the API calls to Consul Servers for you. Perhaps our original perception of how our users want to use Consul on Kubernetes is different from what they expect? I have also seen this request come up before.

3

Hi @david-yu. Thanks for your response.

I suppose our primary use case for Consul, at this time, is Consul KV - for storing application configuration. In future, we may look more closely at the service mesh features of Consul but, for now, Consul KV is our primary area of interest.

At its simplest, what we want to achieve here is:

  • Consul deployed to Kubernetes cluster from Helm chart in a highly available setup.
  • One Consul client per Kubernetes worker node (DaemonSet provides this nicely).
  • Containerized applications deployed to Kubernetes (across multiple namespaces).
  • Applications should be able to reach the Consul KV API over HTTPS/TLS in order to retrieve configuration at startup time.

That last point seems to be tripping me up, as the Consul documentation seems to suggest much of the TLS heavy lifting is automated via Auto-encrypt and the built-in CA. However, doing some rudimentary testing, such as trying to reach the Consul API from a container using wget causes the certificate trust issues as in the original post example. Applications in the container (again, wget for simplicity here) refuses to continue the HTTP connection to the API as such.

Am I misunderstanding how a containerized application in Kubernetes should reach the Consul API over HTTPS? Obviously we don’t want to do skip verification of certs for this.

Let me know if I can provide any additional info - or if you’d like me to clarify anything - would be happy to do so.

Thanks again for your response - Aaron

4

Hi @thecosmicfrog if you’re only talking to Consul servers then you need to use the CA cert in the consul-ca-cert secret. This can be mounted in as a file.

If you want to talk to the local Consul client agent (which is generally recommended for performance) and you have auto encrypt enabled then you need to retrieve a different CA cert from Consul. This is because with auto encrypt enabled the consul clients don’t use the same CA as the servers. We actually have a command: consul-k8s get-consul-client-ca (https://github.com/hashicorp/consul-k8s/tree/master/subcommand/get-consul-client-ca) that makes a call to the Consul servers and retrieves the CA cert and writes it to file. You could use this in an init container in your pod and write the CA cert to a shared volume that gets mounted into your main container.

5

Thanks @lkysow . Is this subcommand referenced anywhere in the Consul documentation?

Also, can you advise if my approach here is correct? i.e. an application running in a Pod accessing Consul KV over HTTPS. Is there a more appropriate approach I should be taking? Thanks!

6

Hey right now there’s only the help docs:

consul-k8s get-consul-client-ca -h
Usage: consul-k8s get-consul-client-ca [options]

  Retrieve Consul client CA certificate by continuously polling
  Consul servers and save it at the provided file location.

HTTP API Options

  -ca-file=<string>
     The path to the CA file to use when making requests to the Consul
     server. This can also be provided via the CONSUL_CACERT environment
     variable instead if preferred. If both values are present, the flag
     value will be used.

  -tls-server-name=<string>
     The server name to set as the SNI header when sending
     HTTPS requests to Consul. This can also be provided via the
     CONSUL_TLS_SERVER_NAME environment variable instead if preferred.
     If both values are present, the flag value will be used.

Command Options

  -log-level=<string>
     Log verbosity level. Supported values (in order of detail) are
     "trace", "debug", "info", "warn", and "error".

  -output-file=<string>
     The file path for writing the Consul client's CA certificate.

  -server-addr=<string>
     The address of the Consul server or the cloud auto-join string. The
     server must be running with TLS enabled. This value is required.

  -server-port=<string>
     The HTTPS port of the Consul server.

We need to do a better job documenting this use-case. This is definitely a valid use-case! You can download the binary from https://releases.hashicorp.com/consul-k8s/ or use our Docker image hashicorp/consul-k8s, see Docker hub for the latest release.

1 Like
7

Thanks @lkysow. Just to clarify, in order to get the Consul client CA cert using the above subcommand of consul-k8s, I need to first retrieve the Consul server CA cert?
As calling the subcommand from one of our application containers results in a CA validation error of its own:

#> consul-k8s get-consul-client-ca -server-addr consul.service.consul -server-port 8501 -output-file /tmp/consul-client-ca.crt

2020-05-27T23:45:43.204Z [ERROR] Error retrieving CA roots from Consul: err="Get "https://consul.service.consul:8501/v1/agent/connect/ca/roots": x509: certificate signed by unknown authority"

Thanks - Aaron

8

Hey Aaron, sorry yes you need the consul-ca-cert because that command talks to the servers.

9

Thanks @lkysow. Something I got working was the following:

  • Create a test Pod YAML file which mounts the Consul server CA cert, as below:
apiVersion: v1
kind: Pod
metadata:
  namespace: default
  name: test-pod
spec:
  volumes:
  - name: consul-consul-ca-cert
    secret:
      secretName: consul-consul-ca-cert
  containers:
  - name: consul-test-pod
    [...]
    volumeMounts:
    - name: consul-consul-ca-cert
      mountPath: /consul/tls/ca
  • When the container comes up, copy the resulting /consul/tls/ca/tls.crt file into /usr/local/share/ca-certificates/consul-server-ca.crt
    • Then execute update-ca-certificates to add the Consul server CA cert as a trusted CA.
  • Test connecting to the Consul server(s) over HTTPS:
#> curl https://consul.service.consul:8501/v1/status/leader
"10.110.2.24:8300"

As you can see, this works! Which is great news. But, this begs the question: If I’m able to connect to the server agent (where KV resides) now without issue, how do I connect to the Consul client to forward my queries?

Happy to provide other examples or further clarification if required.

Thanks - Aaron

10

Nice!

To talk to the Consul client you need to get the client cert using that consul-k8s command and then use the node ip to reach the consul client on the same node, see https://www.consul.io/docs/k8s/installation/overview#accessing-the-consul-http-api.

11

Great, thanks @lkysow. I have proven the below workflow. Perhaps it might help in official documenting of this use case or, at the very least, help others who might stumble upon this post!

  • Create a Kubernetes secret named consul with a key named CONSUL_GOSSIP_ENCRYPTION_KEY and an appropriate encryption key value.
    • Generate value using consul keygen
  • Install the hashicorp/consul Helm chart with an values-override.yaml, such as below:
global:
  datacenter: sandbox

  gossipEncryption:
    secretName: "consul"
    secretKey: "CONSUL_GOSSIP_ENCRYPTION_KEY"

  tls:
    enabled: true
    httpsOnly: true
    enableAutoEncrypt: true
    serverAdditionalDNSSANs: ["'consul.service.consul'"]

server:
  replicas: 3
  bootstrapExpect: 3
  storage: 20Gi

dns:
  clusterIP: 172.20.53.53

ui:
  service:
    type: 'LoadBalancer'

syncCatalog:
  enabled: true
  • Create an example Pod spec to represent our application.
    • Ensure it mounts the Consul server CA cert secret.
    • Ensure the Pod’s container has HOST_IP exposed as an environment variable.
apiVersion: v1
kind: Pod
metadata:
  namespace: default
  name: test-pod
spec:
  volumes:
  - name: consul-consul-ca-cert
    secret:
      secretName: consul-consul-ca-cert
  hostNetwork: false
  containers:
  - name: consul-test-pod
    image: alpine
    imagePullPolicy: IfNotPresent
    env:
    - name: HOST_IP
      valueFrom:
        fieldRef:
          fieldPath: status.hostIP
    command: ["/bin/sh"]
    args: ["-c", "while true; do sleep 24h; done"]
    volumeMounts:
    - name: consul-consul-ca-cert
      mountPath: /consul/tls/ca
  • Upon creation of the Pod, kubectl exec into it, and ensure the ca-certificates and curl packages are installed (I’m using Alpine Linux in this example).
    • (curl is purely for testing purposes)
#> apk update
#> apk add ca-certificates curl
  • Copy the mounted Consul server CA certificate into the /usr/local/share/ca-certificates/ and execute update-ca-certificates to add it to the system root CA store.
#> cp /consul/tls/ca/tls.crt /usr/local/share/ca-certificates/consul-server-ca.crt
#> update-ca-certificates  # might give a trivial warning - ignore it
  • The Consul server is now accessible (and trusted) over HTTPS as below:
#> curl https://consul.service.consul:8501/v1/status/leader
## No TLS errors ##
  • We also want to talk to the Consul client (instead of the server) over HTTPS, for performance reasons.
    • Since the Consul client has its own CA cert, we need to retrieve that from the server.
    • This requires the consul-k8s binary, so we need to get that.
#> cd /usr/local/bin
#> wget https://releases.hashicorp.com/consul-k8s/0.15.0/consul-k8s_0.15.0_linux_amd64.zip  # (or whatever latest version is)
#> unzip consul-k8s_0.15.0_linux_amd64.zip
#> rm consul-k8s_0.15.0_linux_amd64.zip
  • Get the Consul client CA cert and install it via update-ca-certificates:
#> consul-k8s get-consul-client-ca -server-addr consul.service.consul -server-port 8501 -output-file /usr/local/share/ca-certificates/consul-client-ca.crt
#> update-ca-certificates  # might give a trivial warning - ignore it
  • The Consul client is now accessible (and trusted) over HTTPS as below:
#> curl https://$HOST_IP:8501/v1/status/leader
## No TLS errors ##
  • We can also access the Consul KV service from the client without issue:
#> curl https://$HOST_IP:8501/v1/kv/foo/bar/baz
## No TLS errors ##

Naturally, all of the above should be automated by the implementer. These manual steps are purely for demonstration purposes.

I suppose my questions for Hashicorp are:

  1. Is the above considered an appropriate workflow for enabling TLS for Consul and applications in Kubernetes?
  2. Are there any feasible actions Hashicorp would consider implementing to take over some of this heavy lifting?
  3. Most important - can I provide any additional information or feedback in this regard? :slight_smile:

Thanks! - Aaron

12

That’s awesome thanks for the write-up! I’ve created https://github.com/hashicorp/consul-helm/issues/474 to track documenting this.

One thing I’d add is that you could also use an init container that uses the consul-k8s Docker image to run the get-client-ca command and write it to a shared volume. Then you don’t need to download consul-k8s in your image.

13

Thanks @lkysow. Yeah, that would indeed be the preferred approach. The above was a sort of proof-of-concept so I didn’t need to go and write an init container, as that would require some reworking of our assumptions. That’s purely an internal matter though :slight_smile:

Appreciate the creation of the GitHub issue. I’ll track it myself. Glad I could help!