Trouble getting Consul Connect and Envoy to work

Consul
22

Additionally, Consul itself doesn’t support (or at least not well?) including the chain in the -client-cert=... or cert_file = ... parameters (ie. having ca_file = ... file include only the root), as you’ll either get key doesn't match cert or simply the ambiguous TLS/connection dropped/untrusted CA errors, so it’s best to have the full chain in the files for -ca-file=.../ca_path = ...

1 Like
23

I am using internal Consul CA.

As per docs: https://www.consul.io/docs/connect/ca/consul

Its asking to generate SPIFFE CA signing Cert and add the cluster id with .consul TLD .

How do we do that ?

24

Lastly, I use a variation of a script I’ve written for openssl root CA; feel free to base yours off this, but you will need to make modifications and double check things, as I hastily redacted a few things specific to my company off of it:

#!/bin/bash -xe

set -xe

export DEBIAN_FRONTEND=noninteractive

export PASSPHRASE=$(dd if=/dev/urandom bs=1M count=64 2>&1 | sha1sum | tail -1 | awk '{print $1}')

echo "ROOT CA KEY PASSPHRASE: $PASSPHRASE"

apt-get update
apt-get install -y openssl

touch /root/.rnd

mkdir -p /root/ca
mkdir -p /root/ca/certs
mkdir -p /root/ca/crl
mkdir -p /root/ca/csr
mkdir -p /root/ca/newcerts
mkdir -p /root/ca/private
chmod 700 /root/ca/private
touch /root/ca/index.txt
echo 1000 > /root/ca/serial

cat > /root/ca/index.txt.attr <<EOF
unique_subject = no
EOF

cp /etc/ssl/openssl.cnf /root/ca/openssl.cnf

sed -i '
/^\[ CA_default \]$/,/\[/ {
    s^\(dir\s\+=\) ./demoCA^\1 /root/ca^
    s/#\(unique_subject\s\+=\s\+no\)/\1/
    /^policy\s\+=\s\+policy_match$/ a\email_in_dn\t= yes
}

/^\[ req \]$/,/^\[/ {
    s/\(default_bits\s\+=\) 2048/\1 4096/
}

/^\[ v3_ca \]$/,/\[/ {
    s/\(basicConstraints =\).*/\1 critical, CA:true/
    /^# subjectAltName=email:copy$/ a\subjectAltName=${ENV::SAN}
}

/^\[ usr_cert \]$/,/\[/ {
    /^# subjectAltName=email:move$/ a\subjectAltName=${ENV::SAN}
}
' /root/ca/openssl.cnf

CONFIG=$(sed '/\[ v3_ca \]/!d; :loop; n; /\[/d; $!bloop' /root/ca/openssl.cnf)

echo "$CONFIG" | sed '
    s/^\[ v3_ca \]$/[ v3_intermediate_ca ]/
    s/# \(keyUsage =\).*/\1 cRLSign, keyCertSign, digitalSignature, keyEncipherment/
    s/\(basicConstraints =\).*/\1 critical, CA:true, pathlen:2/
    /^keyUsage\s\+=\s\+/ a\extendedKeyUsage = serverAuth, clientAuth
' >> /root/ca/openssl.cnf

openssl genpkey \
            -out /root/ca/private/ca.key.pem \
            -outform PEM \
            -pass pass:$PASSPHRASE \
            -algorithm ec \
            -pkeyopt ec_paramgen_curve:secp521r1 \
            -aes256
chmod 400 /root/ca/private/ca.key.pem

export SAN="DNS:localhost, IP:127.0.0.1, DNS:ca.example.internal"

openssl req -config /root/ca/openssl.cnf \
        -passin=pass:$PASSPHRASE \
        -key /root/ca/private/ca.key.pem \
        -new \
        -x509 \
        -days 3650 \
        -sha256 \
        -extensions v3_ca \
        -out /root/ca/certs/ca.cert.pem \
        -subj "/emailAddress=devops@example.network/C=CA/ST=British Columbia/L=Vancouver/O=Example/OU=ca.example.internal/CN=ca.example.internal"

ln -s /root/ca/private/ca.key.pem /root/ca/private/cakey.pem
ln -s /root/ca/certs/ca.cert.pem /root/ca/cacert.pem

new_intermediate_keypair() {

    HOSTNAME=$1
    DATACENTER=$2
    DOMAIN=$3
    COMMON_NAME=${HOSTNAME}.${DATACENTER}.${DOMAIN}
    SPIFFE=spiffe://3116ff7e-e927-f54b-6ef6-1a7943cbb49e.consul

    openssl genpkey \
            -out /root/ca/private/${COMMON_NAME}.key.pem \
            -outform PEM \
            -pass pass:$PASSPHRASE \
            -algorithm ec \
            -pkeyopt ec_paramgen_curve:secp521r1 \
            -aes256

    chmod 400 /root/ca/private/${COMMON_NAME}.key.pem

    export SAN="DNS:localhost, IP:127.0.0.1, URI:${SPIFFE}, DNS:${COMMON_NAME}"

    openssl req -config /root/ca/openssl.cnf \
            -passin=pass:$PASSPHRASE \
            -key /root/ca/private/${COMMON_NAME}.key.pem \
            -new -sha256 \
            -out /root/ca/csr/${COMMON_NAME}.csr.pem \
            -subj "/emailAddress=devops@example.internal/C=CA/ST=British Columbia/L=Vancouver/O=Example/OU=${COMMON_NAME}/CN=${COMMON_NAME}"

    openssl ca -config /root/ca/openssl.cnf \
            -passin=pass:$PASSPHRASE \
            -days 365 \
            -notext \
            -md sha256 \
            -extensions v3_intermediate_ca \
            -in /root/ca/csr/${COMMON_NAME}.csr.pem \
            -out /root/ca/certs/${COMMON_NAME}.cert.pem \
            -batch

    chmod 644 /root/ca/certs/${COMMON_NAME}.cert.pem

    openssl ec -in /root/ca/private/${COMMON_NAME}.key.pem \
            -passin=pass:$PASSPHRASE \
            -out /root/ca/private/${COMMON_NAME}.key.plaintext.pem # remove passphrase (for consul/vault)
}

new_intermediate_keypair vault dc1 consul
1 Like
25

Cluster ID comes from https://www.consul.io/api/connect/ca#list-ca-root-certificates /connect/ca/roots, and it needs to be set in the SAN; I use a variation of this to do so:

cat /etc/ssl/openssl.cnf
...
[ v3_ca ]
...
subjectAltName=${ENV::SAN}
...

and when generating the CSR:

export SAN="DNS:localhost, IP:127.0.0.1, URI:spiffe://___CLUSTER_ID___.consul, DNS:server.dc1.consul"
openssl req -config /etc/ssl/openssl.cnf ...
1 Like
28

Hi @quinndiggity,

I generated the consul server certificates using the intermediate CA. I get this.

openssl verify -CAfile custom_root_CA.pem custom_intermediate_CA.pem dc1-server-consul-0.pem

custom_intermediate_CA.pem: OK
CN = server.dc1.consul
error 20 at 0 depth lookup: unable to get local issuer certificate
error dc1-server-consul-0.pem: verification failed

openssl verify -CAfile custom_root_CA.pem -untrusted custom_intermediate_CA.pem dc1-server-consul-0.pem

dc1-server-consul-0.pem: OK

and I use the same intermediate CA to generate consul client certificate… on applying the respective certificate on the consul server and the consul client and starting the envoy
proxy …I get this error in the envoy logs. (exposed the environment variable to inject consul cafile, cert, key, consul http addr, grpc addr and http ssl as true)

[WARN] agent: grpc: Server.Serve failed to complete security handshake from “127.0.0.1:42558”: remote error: tls: unknown certificate authority

what can be wrong here ?

Hi @quinndiggity,

I fixed the above issue by concatenating the contents of Root CA and Intermediate CA to form a CA chain and the chain validation looks for good both(server and client) for the certificates created from this intermediate CA .

However, After loading these certs into the consul server and client and passing the concatenated contents of Root CA and Intermediate CA (CA chain) via a -ca-file parameter

to envoy proxy… it gives me this error:

Error adding/updating listener(s) public_listener:0.0.0.0:21000: Failed to load trusted CA certificates from

What can be wrong here ? How are you currently doing the setup ?

Hi @quinndiggity,

So finally I figured it out. I was placing the CA certs (root and intermediate) in wrong order.
Placing the certs in correct order(intermediate CA cert first followed by root CA cert) in custom_intermediate_CA.pem file and creating consul server and consul client certs from this intermediate cert fixed my envoy communication issue.

Thanks @quinndiggity for all your help. Much appreciated.