Consul verified TLS from Pods in Kubernetes cluster

Consul
11

Great, thanks @lkysow. I have proven the below workflow. Perhaps it might help in official documenting of this use case or, at the very least, help others who might stumble upon this post!

  • Create a Kubernetes secret named consul with a key named CONSUL_GOSSIP_ENCRYPTION_KEY and an appropriate encryption key value.
    • Generate value using consul keygen
  • Install the hashicorp/consul Helm chart with an values-override.yaml, such as below:
global:
  datacenter: sandbox

  gossipEncryption:
    secretName: "consul"
    secretKey: "CONSUL_GOSSIP_ENCRYPTION_KEY"

  tls:
    enabled: true
    httpsOnly: true
    enableAutoEncrypt: true
    serverAdditionalDNSSANs: ["'consul.service.consul'"]

server:
  replicas: 3
  bootstrapExpect: 3
  storage: 20Gi

dns:
  clusterIP: 172.20.53.53

ui:
  service:
    type: 'LoadBalancer'

syncCatalog:
  enabled: true
  • Create an example Pod spec to represent our application.
    • Ensure it mounts the Consul server CA cert secret.
    • Ensure the Pod’s container has HOST_IP exposed as an environment variable.
apiVersion: v1
kind: Pod
metadata:
  namespace: default
  name: test-pod
spec:
  volumes:
  - name: consul-consul-ca-cert
    secret:
      secretName: consul-consul-ca-cert
  hostNetwork: false
  containers:
  - name: consul-test-pod
    image: alpine
    imagePullPolicy: IfNotPresent
    env:
    - name: HOST_IP
      valueFrom:
        fieldRef:
          fieldPath: status.hostIP
    command: ["/bin/sh"]
    args: ["-c", "while true; do sleep 24h; done"]
    volumeMounts:
    - name: consul-consul-ca-cert
      mountPath: /consul/tls/ca
  • Upon creation of the Pod, kubectl exec into it, and ensure the ca-certificates and curl packages are installed (I’m using Alpine Linux in this example).
    • (curl is purely for testing purposes)
#> apk update
#> apk add ca-certificates curl
  • Copy the mounted Consul server CA certificate into the /usr/local/share/ca-certificates/ and execute update-ca-certificates to add it to the system root CA store.
#> cp /consul/tls/ca/tls.crt /usr/local/share/ca-certificates/consul-server-ca.crt
#> update-ca-certificates  # might give a trivial warning - ignore it
  • The Consul server is now accessible (and trusted) over HTTPS as below:
#> curl https://consul.service.consul:8501/v1/status/leader
## No TLS errors ##
  • We also want to talk to the Consul client (instead of the server) over HTTPS, for performance reasons.
    • Since the Consul client has its own CA cert, we need to retrieve that from the server.
    • This requires the consul-k8s binary, so we need to get that.
#> cd /usr/local/bin
#> wget https://releases.hashicorp.com/consul-k8s/0.15.0/consul-k8s_0.15.0_linux_amd64.zip  # (or whatever latest version is)
#> unzip consul-k8s_0.15.0_linux_amd64.zip
#> rm consul-k8s_0.15.0_linux_amd64.zip
  • Get the Consul client CA cert and install it via update-ca-certificates:
#> consul-k8s get-consul-client-ca -server-addr consul.service.consul -server-port 8501 -output-file /usr/local/share/ca-certificates/consul-client-ca.crt
#> update-ca-certificates  # might give a trivial warning - ignore it
  • The Consul client is now accessible (and trusted) over HTTPS as below:
#> curl https://$HOST_IP:8501/v1/status/leader
## No TLS errors ##
  • We can also access the Consul KV service from the client without issue:
#> curl https://$HOST_IP:8501/v1/kv/foo/bar/baz
## No TLS errors ##

Naturally, all of the above should be automated by the implementer. These manual steps are purely for demonstration purposes.

I suppose my questions for Hashicorp are:

  1. Is the above considered an appropriate workflow for enabling TLS for Consul and applications in Kubernetes?
  2. Are there any feasible actions Hashicorp would consider implementing to take over some of this heavy lifting?
  3. Most important - can I provide any additional information or feedback in this regard? :slight_smile:

Thanks! - Aaron