Hi everyone,
I’m confused with managing Consuls internal CA and cert distribution.
From what I know, to enable TLS in consul (using self-signed certificates generated by Consul), the first step is to generate a CA certificate and keyfile using the consul tls ca create command. This creates two files:
- consul-agent-ca.pem
- consul-agent-ca-key.pem
According to the Deployment Guide, the first file (consul-agent-ca.pem) is used with the ca_file parameter and needs to be copied over to all agents in order to enable all agents to verify certificates issued by this CA.
But when I try to use the HTTPS interface to query the API, I receive an error:
> consul catalog services
Error listing services: Get "https://127.0.0.1:8501/v1/catalog/services": x509: certificate signed by unknown authority
> set | grep CONSUL_CACERT
CONSUL_CACERT=/etc/consul.d/CA/consul-agent-ca.pem
> openssl x509 -text -in ${CONSUL_CACERT}
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
19:58:6f:c9:52:f3:63:58:e8:a3:5c:56:42:99:98:22
Signature Algorithm: ecdsa-with-SHA256
Issuer: C = US, ST = CA, L = San Francisco, street = 101 Second Street, postalCode = 94105, O = HashiCorp Inc., CN = Consul Agent CA 33689889316172641290330051500966516770
Validity
Not Before: Aug 18 16:29:01 2022 GMT
Not After : Aug 17 16:29:01 2027 GMT
Subject: C = US, ST = CA, L = San Francisco, street = 101 Second Street, postalCode = 94105, O = HashiCorp Inc., CN = Consul Agent CA 33689889316172641290330051500966516770
Subject Public Key Info:
Public Key Algorithm: id-ecPublicKey
Public-Key: (256 bit)
pub:
04:ff:b5:4b:8f:07:32:9f:ac:87:f3:bd:e6:aa:51:
11:85:8e:61:65:a2:ee:2a:11:58:26:7e:6d:0b:3e:
78:5a:a9:77:5c:fa:36:4b:87:89:14:82:66:07:de:
d3:a9:89:1f:8a:7d:e0:ef:09:a2:08:8e:4c:4f:49:
63:96:33:51:01
ASN1 OID: prime256v1
NIST CURVE: P-256
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Certificate Sign, CRL Sign
X509v3 Basic Constraints: critical
CA:TRUE
X509v3 Subject Key Identifier:
D5:49:88:B5:5B:5C:CA:26:53:1B:B7:85:E7:76:1B:CE:B9:B8:A2:6B:02:01:B3:C1:5A:8C:3F:84:D2:B4:94:C4
X509v3 Authority Key Identifier:
keyid:D5:49:88:B5:5B:5C:CA:26:53:1B:B7:85:E7:76:1B:CE:B9:B8:A2:6B:02:01:B3:C1:5A:8C:3F:84:D2:B4:94:C4
Signature Algorithm: ecdsa-with-SHA256
30:44:02:20:6b:78:23:d6:a5:b4:4a:45:97:fb:7e:29:a8:bf:
eb:f9:76:81:0c:23:f0:c2:f5:d5:fd:a4:12:b5:d5:63:4b:21:
02:20:28:04:a5:1f:68:a4:a9:5b:00:ea:76:82:ce:95:d7:c7:
0f:db:53:68:9a:e1:75:f6:dd:ff:ac:23:35:33:54:21
-----BEGIN CERTIFICATE-----
MIIC6jCCApGgAwIBAgIQGVhvyVLzY1joo1xWQpmYIjAKBggqhkjOPQQDAjCBuDEL
MAkGA1UEBhMCVVMxCzAJBgNVBAgTAkNBMRYwFAYDVQQHEw1TYW4gRnJhbmNpc2Nv
MRowGAYDVQQJExExMDEgU2Vjb25kIFN0cmVldDEOMAwGA1UEERMFOTQxMDUxFzAV
BgNVBAoTDkhhc2hpQ29ycCBJbmMuMT8wPQYDVQQDEzZDb25zdWwgQWdlbnQgQ0Eg
MzM2ODk4ODkzMTYxNzI2NDEyOTAzMzAwNTE1MDA5NjY1MTY3NzAwHhcNMjIwODE4
MTYyOTAxWhcNMjcwODE3MTYyOTAxWjCBuDELMAkGA1UEBhMCVVMxCzAJBgNVBAgT
AkNBMRYwFAYDVQQHEw1TYW4gRnJhbmNpc2NvMRowGAYDVQQJExExMDEgU2Vjb25k
IFN0cmVldDEOMAwGA1UEERMFOTQxMDUxFzAVBgNVBAoTDkhhc2hpQ29ycCBJbmMu
MT8wPQYDVQQDEzZDb25zdWwgQWdlbnQgQ0EgMzM2ODk4ODkzMTYxNzI2NDEyOTAz
MzAwNTE1MDA5NjY1MTY3NzAwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAAT/tUuP
BzKfrIfzveaqURGFjmFlou4qEVgmfm0LPnhaqXdc+jZLh4kUgmYH3tOpiR+KfeDv
CaIIjkxPSWOWM1EBo3sweTAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUwAwEB
/zApBgNVHQ4EIgQg1UmItVtcyiZTG7eF53Ybzrm4omsCAbPBWow/hNK0lMQwKwYD
VR0jBCQwIoAg1UmItVtcyiZTG7eF53Ybzrm4omsCAbPBWow/hNK0lMQwCgYIKoZI
zj0EAwIDRwAwRAIga3gj1qW0SkWX+34pqL/r+XaBDCPwwvXV/aQStdVjSyECICgE
pR9opKlbAOp2gs6V18cP21NomuF19t3/rCM1M1Qh
-----END CERTIFICATE-----
Following a hint from Discuss user @laukaichung in another thread, one must grab the HTTP cert via HTTP, store it somewhere and set CONSUL_CACERT pointing to that file, which differs from what’s in the file /etc/consul.d/CA/consul-agent-ca.pem from CA generation:
> curl -k http://127.0.0.1:8500/v1/connect/ca/roots | jq -r '.Roots[]."RootCert"' >> /ca.cert
> openssl x509 -text -in /ca.cert
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 9 (0x9)
Signature Algorithm: ecdsa-with-SHA256
Issuer: CN = pri-1v38s674.consul.ca.130259ea.consul
Validity
Not Before: Aug 18 16:29:22 2022 GMT
Not After : Aug 15 16:29:22 2032 GMT
Subject: CN = pri-1v38s674.consul.ca.130259ea.consul
Subject Public Key Info:
Public Key Algorithm: id-ecPublicKey
Public-Key: (256 bit)
pub:
04:ab:7e:cd:8a:f4:c7:b5:0b:0d:94:46:c2:73:4a:
ff:23:60:1f:ac:43:57:5a:d1:03:d4:88:c2:a8:2d:
1f:ee:f3:61:2d:d1:32:13:a2:8e:81:80:39:08:1f:
18:29:d4:ae:73:a6:a6:00:51:d6:7c:1d:1b:c7:0b:
b2:02:4c:63:75
ASN1 OID: prime256v1
NIST CURVE: P-256
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Certificate Sign, CRL Sign
X509v3 Basic Constraints: critical
CA:TRUE
X509v3 Subject Key Identifier:
7E:6B:D2:C4:75:0E:BE:8F:BB:24:8C:77:B5:50:A9:CB:79:8E:48:03:2E:7F:A6:3F:44:E2:A5:6E:65:22:3E:F3
X509v3 Authority Key Identifier:
keyid:7E:6B:D2:C4:75:0E:BE:8F:BB:24:8C:77:B5:50:A9:CB:79:8E:48:03:2E:7F:A6:3F:44:E2:A5:6E:65:22:3E:F3
X509v3 Subject Alternative Name:
URI:spiffe://130259ea-c3ab-37d1-abf1-39af49c537fc.consul
Signature Algorithm: ecdsa-with-SHA256
30:46:02:21:00:d0:5f:c9:51:ce:dd:11:b5:22:c2:e2:65:dd:
7c:50:93:b9:8c:0e:79:b5:1a:86:8c:67:50:83:7a:6f:84:94:
7e:02:21:00:b9:15:c9:21:92:5b:b2:d9:27:5c:fe:47:3f:95:
ef:40:85:7b:e2:45:a7:80:bf:26:f6:8d:34:bb:a7:e0:cc:6b
-----BEGIN CERTIFICATE-----
MIICEDCCAbWgAwIBAgIBCTAKBggqhkjOPQQDAjAxMS8wLQYDVQQDEyZwcmktMXYz
OHM2NzQuY29uc3VsLmNhLjEzMDI1OWVhLmNvbnN1bDAeFw0yMjA4MTgxNjI5MjJa
Fw0zMjA4MTUxNjI5MjJaMDExLzAtBgNVBAMTJnByaS0xdjM4czY3NC5jb25zdWwu
Y2EuMTMwMjU5ZWEuY29uc3VsMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEq37N
ivTHtQsNlEbCc0r/I2AfrENXWtED1IjCqC0f7vNhLdEyE6KOgYA5CB8YKdSuc6am
AFHWfB0bxwuyAkxjdaOBvTCBujAOBgNVHQ8BAf8EBAMCAYYwDwYDVR0TAQH/BAUw
AwEB/zApBgNVHQ4EIgQgfmvSxHUOvo+7JIx3tVCpy3mOSAMuf6Y/ROKlbmUiPvMw
KwYDVR0jBCQwIoAgfmvSxHUOvo+7JIx3tVCpy3mOSAMuf6Y/ROKlbmUiPvMwPwYD
VR0RBDgwNoY0c3BpZmZlOi8vMTMwMjU5ZWEtYzNhYi0zN2QxLWFiZjEtMzlhZjQ5
YzUzN2ZjLmNvbnN1bDAKBggqhkjOPQQDAgNJADBGAiEA0F/JUc7dEbUiwuJl3XxQ
k7mMDnm1GoaMZ1CDem+ElH4CIQC5Fckhkluy2Sdc/kc/le9AhXviRaeAvyb2jTS7
p+DMaw==
-----END CERTIFICATE-----
From here, I am totaly puzzled which cert I need to set for tls.defaults.ca_file / ${CONSUL_CACERT}. I mean - obviously I know my Tools and how to extract whatever certificate from whereever, but I have no idea how I need to setup the TLS parts here to make them work. Consul acts as if I am dealing with completely different TLS chains here, even though I followed the Deployment guide on this and the CA file I configured (and even added to /usr/local/share/ca-certificates/consul_ca_root_certs.crt and ran update-ca-certificates) seems to be the correct one to verify the authenticity of the HTTPS API just fine:
> set | grep CONSUL_CACERT
CONSUL_CACERT=/etc/consul.d/CA/consul-agent-ca.pem
> openssl verify -verbose -CAfile ${CONSUL_CACERT} /ca.cert
/ca.cert: OK
Can someone tell me what I need to do to have working Auto Encryption and a working Consul CA that issues server certificates other from what I have done already?
Best regards,
Marc