I am setting up consul with TLS and auto_config, which seems to have been a project I could not have attempted at a worse time, considering the grpc issues that were solved in 1.5.0, due to enormous confusion about where the problem has been laying. I am currently FINALLY at the stage where I believe I understand my problem, although, I am still a bit confused about why it occurs. In short, when setting up my upstream my certificate chain looks like this:
# openssl s_client -showcerts -connect 10.0.0.6:27106
CONNECTED(00000003)
Can't use SSL_get_servername
depth=1 CN = pri-9g3tg0w.vault.ca.5e245593.consul
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0
verify return:1
---
Certificate chain
0 s:
i:CN = pri-9g3tg0w.vault.ca.5e245593.consul
a:PKEY: id-ecPublicKey, 256 (bit); sigalg: RSA-SHA256
v:NotBefore: Mar 20 17:51:18 2023 GMT; NotAfter: Mar 20 18:51:48 2023 GMT
-----BEGIN CERTIFICATE-----
MIICyTCCAbGgAwIBAgIUUPq4hVx9BON3TZMu69ErkvQx2g8wDQYJKoZIhvcNAQEL
BQAwLzEtMCsGA1UEAxMkcHJpLTlnM3RnMHcudmF1bHQuY2EuNWUyNDU1OTMuY29u
c3VsMB4XDTIzMDMyMDE3NTExOFoXDTIzMDMyMDE4NTE0OFowADBZMBMGByqGSM49
AgEGCCqGSM49AwEHA0IABGB5iOntBnuHDiy1IDsXb6QbvhGqyc0sDVmHe8fxKloK
/NrpGkgQ7uNQXhhlQ28qwZParTzntCWp4RzxIhvmpA6jgdYwgdMwDgYDVR0PAQH/
BAQDAgOoMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAdBgNVHQ4EFgQU
h1HIi13CBDmWJ3rbnMo2KvipDKcwHwYDVR0jBBgwFoAUdMRDc29u85fU/Ze4IokC
3FEoVfUwYgYDVR0RAQH/BFgwVoZUC3BpZmZlOi8vNWUyNDU1OTMtNjUyYS00NDhh
LThjNzgtYWUxYTFkOTMwNjUzLmNvbnN1bC9ucy9kZWZhdWx0L2RjL2RjMS9zdmMv
Y291bnQtYXBpMA0GCSqGSIb3DQEBCwUAA4IBAQCOANBaJOdx33BHKddGXu9FnnFs
C6LkhYlFWzvPx7H4xn6775LJFCkder9AZyYj1FE3rLw+Z1A6pspTTe1xR8LSiVGc
VmtHsE1M5KIXuIEBgRh7lAhcjf4CMcy/HsjRjI6bG3J04S2FAm3Swl6mTz5LoDjC
GJ5PW4zy9Hl+uxFihURHVizROpAeNV+qZIeSyzI1NabGtNutbLJjKhTM6FUD1aam
XFi/NYcQaLQqLXlb2YnOGc+GkgdB9LvhyJkyG/FuQLFFBJgHGpNrGXVrPVcbkYmP
3uD14g5vAaZTXlrYG42HZ4NMZERXQybwzM0U3q7QLk5qOJKQS0No5CrtgVut
-----END CERTIFICATE-----
1 s:CN = pri-9g3tg0w.vault.ca.5e245593.consul
i:CN = fnurf.bootstrap
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Mar 20 17:46:18 2023 GMT; NotAfter: Mar 21 17:46:48 2023 GMT
-----BEGIN CERTIFICATE-----
MIIDrzCCApegAwIBAgIUZ/HCG/tXlzAqmvAZV3f7pTBc8ZcwDQYJKoZIhvcNAQEL
BQAwJjEkMCIGA1UEAxMbaGV0em5lci5hZGl0cm9uaWMuYm9vdHN0cmFwMB4XDTIz
MDMyMDE3NDYxOFoXDTIzMDMyMTE3NDY0OFowLzEtMCsGA1UEAxMkcHJpLTlnM3Rn
MHcudmF1bHQuY2EuNWUyNDU1OTMuY29uc3VsMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEAoNxpUhpirj4MLWmy8LIBr++Hw+Qo+HwmfoCb1re0mDAcMmKI
ThVwcthkg8WbE47NEz9WciqyZ43i3t1e96hIu9vLBSvFoNJQMnDVWUl7BzXcSLZQ
hG43Nnd00ccXA35Gy23Ayvx+RkfdIJKyUfJVrsMNYzDO/dJ0NTehmrXhxs/hx/0d
H6GwGo/9iR2joODEheECLFG2EKf7vJefan2p/BL2hUd595LFQQ7B01NRWe00hIAt
yY8JRk7G9iONe2OE4JgSDAQtzBgM5bklxf0OyNVlkcJ6UWyveSGd0ztiqUCnboVH
/wMMb7sDrfGp6GIwvTn4eh/Fx7XBVayG5+3+bwIDAQABo4HLMIHIMA4GA1UdDwEB
/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBR0xENzb27zl9T9l7gi
iQLcUShV9TAfBgNVHSMEGDAWgBRyTn/FLOdDrAbdg6n6YCompVa8ejBlBgNVHREE
XjBcgiRwcmktOWczdGcwdy52YXVsdC5jYS41ZTI0NTU5My5jb25zdWyGNHNwaWZm
ZTovLzVlMjQ1NTkzLTY1MmEtNDQ4YS04Yzc4LWFlMWExZDkzMDY1My5jb25zdWww
DQYJKoZIhvcNAQELBQADggEBAELY4raK3Ub8sfzCRgXsrptROV+qI+2p3PZE3icb
a3tF6pKJwupvGGaohzY0p41nvScFf91Uwz4t1Y3X6buTQi6V0htpSGU8pKjG2TH8
EgIQhAaeIP2GNwaHmUYv9x8fgNmO9dM39PSObFR+VSqYamVt83fNj7ESSMKXFkU7
OPF2FNeMBkULCvsrxmXkaL+eGWqNkyMLBX7NvUky+sgKY3wN/o4vbsnIQCanlGsU
QLzIzYHtO0K6Oq/o8C21kNDbHMeoObr6FMWDfBMsqIdXrmiYoF4Mb72mbHqTvfLp
p9yQjWBQWiTVnvInRK7JnBr5keGSGyNy3TMJMLguXw1J2Lg=
-----END CERTIFICATE-----
---
Server certificate
subject=
issuer=CN = pri-9g3tg0w.vault.ca.5e245593.consul
---
Acceptable client certificate CA names
CN = fnurf.bootstrap
CN = fnurf.internal
Requested Signature Algorithms: ECDSA+SHA256:RSA-PSS+SHA256:RSA+SHA256:ECDSA+SHA384:RSA-PSS+SHA384:RSA+SHA384:RSA-PSS+SHA512:RSA+SHA512:0x01+0x02
Shared Requested Signature Algorithms: ECDSA+SHA256:RSA-PSS+SHA256:RSA+SHA256:ECDSA+SHA384:RSA-PSS+SHA384:RSA+SHA384:RSA-PSS+SHA512:RSA+SHA512
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2087 bytes and written 405 bytes
Verification error: unable to get local issuer certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 20 (unable to get local issuer certificate)
---
80BBED69DC7F0000:error:0A00045C:SSL routines:ssl3_read_bytes:tlsv13 alert certificate required:ssl/record/rec_layer_s3.c:1584:SSL alert number 116
So, despite vault returning the following ca_chain:
# vault read -format=json consul_connect/issuer/default | jq -r .data.ca_chain[]
-----BEGIN CERTIFICATE-----
MIIDrzCCApegAwIBAgIUZ/HCG/tXlzAqmvAZV3f7pTBc8ZcwDQYJKoZIhvcNAQEL
BQAwJjEkMCIGA1UEAxMbaGV0em5lci5hZGl0cm9uaWMuYm9vdHN0cmFwMB4XDTIz
MDMyMDE3NDYxOFoXDTIzMDMyMTE3NDY0OFowLzEtMCsGA1UEAxMkcHJpLTlnM3Rn
MHcudmF1bHQuY2EuNWUyNDU1OTMuY29uc3VsMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEAoNxpUhpirj4MLWmy8LIBr++Hw+Qo+HwmfoCb1re0mDAcMmKI
ThVwcthkg8WbE47NEz9WciqyZ43i3t1e96hIu9vLBSvFoNJQMnDVWUl7BzXcSLZQ
hG43Nnd00ccXA35Gy23Ayvx+RkfdIJKyUfJVrsMNYzDO/dJ0NTehmrXhxs/hx/0d
H6GwGo/9iR2joODEheECLFG2EKf7vJefan2p/BL2hUd595LFQQ7B01NRWe00hIAt
yY8JRk7G9iONe2OE4JgSDAQtzBgM5bklxf0OyNVlkcJ6UWyveSGd0ztiqUCnboVH
/wMMb7sDrfGp6GIwvTn4eh/Fx7XBVayG5+3+bwIDAQABo4HLMIHIMA4GA1UdDwEB
/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBR0xENzb27zl9T9l7gi
iQLcUShV9TAfBgNVHSMEGDAWgBRyTn/FLOdDrAbdg6n6YCompVa8ejBlBgNVHREE
XjBcgiRwcmktOWczdGcwdy52YXVsdC5jYS41ZTI0NTU5My5jb25zdWyGNHNwaWZm
ZTovLzVlMjQ1NTkzLTY1MmEtNDQ4YS04Yzc4LWFlMWExZDkzMDY1My5jb25zdWww
DQYJKoZIhvcNAQELBQADggEBAELY4raK3Ub8sfzCRgXsrptROV+qI+2p3PZE3icb
a3tF6pKJwupvGGaohzY0p41nvScFf91UWz4t1Y3X6buTQi6V0htpSGU8pKjG2TH8
EgIqhAaeIP2GNwaHmUYv9x8fgNmO9dM39PSObFR+VSqYamVt83fNj7ESSMKXFkU7
OPF2FNeMBkULCvsrxmXkaL+eGWqNkyMLBX7NvUky+sgKY3wN/o4vbsnIQCanlGsU
QLzIzYHtO0K6Oq/o8C21kNDbHMeoObr6FMWDfBMsqIdXrmiYoF4Mb72mbHqTvfLp
p9yQjWBQWiTVnvInRK7JnBr5keGSGyNy3TMJMLguXw1J2Lg=
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIDzjCCAragAwIBAgIUVJMtsCfNYM/A4sklQSXuqoPC2LkwDQYJKoZIhvcNAQEL
BQAwHTEbMBkGA1UEAxMSYWRpdHJvbmljLmludGVybmFsMB4XDTIzMDIyNTE4MDQ0
MloXDTIzMDQzMDIzNTk1OVowJjEkMCIGA1UEAxMbaGV0em5lci5hZGl0cm9uaWMu
Ym9vdHN0cmFwMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA3o5A8inR
sY27GtxjCXFKsrWmHNmW7fuj7dBW84IgMqAQW3rwmXi7qh4fADvaP/vV1eoyJCdJ
4ioQ5fsOiw7/UFlDT3qAqaiIeKpMfmM1H8e4yjAX3Hz3bhDuPW/JvWc6UcLg/5KU
AkfUjjsDezsrof73isAfS8TTyBVsZHErKHp0r/uP3K1uScOOkpr6DcsImaK8e4k+
GNScVht5cJNshq7ImDv8sNyrIJdpkqSFuzom3lb6jY0ICAPPm0BTkCociOlM2Cxb
eY3KMWkgdhWcDBWqnYQhHQskmlOWml5zxy9MzirDWVRk9f7WXoi2CYBlE4HL2HJW
IEnXHgOw7FSFGwIDAQABo4H8MIH5MA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8E
BTADAQH/MB0GA1UdDgQWBBRyTn/FLOdDrAbdg6n6YCompVa8ejAfBgNVHSMEGDAW
gBT6O8D7yaP3dNSYLONXtjfKvIgvbzA7BggrBgEFBQcBAQQvMC0wKwYIKwYBBQUH
MAKGH2h0dHA6Ly8xMjcuMC4wLjE6ODIwMC92MS9wa2kvY2EwJgYDVR0RBB8wHYIb
aGV0em5lci5hZGl0cm9uaWMuYm9vdHN0cmFwMDEGA1UdHwQqMCgwJqAkoCKGIGh0
dHA6Ly8xMjcuMC4wLjE6ODIwMC92MS9wa2kvY3JsMA0GCSqGSIb3DQEBCwUAA4IB
AQCnZM2tUXzdWbUbzQnFOjbTMmYeJfD8W1scoN6r560fahEfZVjRHLgniVeNATGw
vEuyKFJzraHsVBiiFGsIpG+F+ojMJvg52YbIgwpzuS3M/JC+ce/hELSWjdIl4jB0
PM6na0bcS2u0pEaR9B7xm8F3mphicAntx2b6M/ESy725cQXK4diQt91ebR8Futtg
wBdDWituttDQyHdv1n9x1rfNlEEodBLiaIXExuEJXtB9o3b+druYcz/FKiT39q3n
A0tDSRIPyZwF7eTnK+I+0OtzjOs9R6GXbbJmPHBKewj4NmjcunxR2e+zNkQzLI1k
TS/NGkSHwkLSfs3RB2xF76WB
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIDvDCCAqSgAwIBAgIUG8heeXQq5yDRheUi96nK5A8PI4AwDQYJKoZIhvcNAQEL
BQAwHTEbMBkGA1UEAxMSYWRpdHJvbmljLmludGVybmFsMB4XDTIzMDIyNTE3NDky
MVoXDTIzMTIzMTIzNTk1OVowHTEbMBkGA1UEAxMSYWRpdHJvbmljLmludGVybmFs
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzrak3vB8JjOnDDGOpabo
+/KFyWTd/WtW4LYzfO9kq0Y+r14/DYejL8zNsqvXI0xTVHgJvcKcwgc7p2U1gMAE
6U36GJZXJeUMG9BB+Ifbkd8B5WN3Ro0kilFxYZFAGpxA4awsNzYHejXctSEJj036
nVZhddd2m0CreUMhLVr38jm0sHe7KXaTIj476x5m+gkoFyvhV0WXQbvGfNnEWT9K
rJYTgn5Q8+XCHgjkM34XDtenSOzPT9o8dEXxKZKu68T9XfJ9ZV4BlvBFeoe5GvYj
k5Hw/ipaHCB+A7gb91iiqQg98Byow5nX2EG7kWoKMwTkT9sr70OOyVedBT5xzETl
dwIDAQABo4HzMIHwMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MB0G
A1UdDgQWBBT6O8D7yaP3dNSYLONXtjfKvIgvbzAfBgNVHSMEGDAWgBT6O8D7yaP3
dNSYLONXtjfKvIgvbzA7BggrBgEFBQcBAQQvMC0wKwYIKwYBBQUHMAKGH2h0dHA6
Ly8xMjcuMC4wLjE6ODIwMC92MS9wa2kvY2EwHQYDVR0RBBYwFIISYWRpdHJvbmlj
LmludGVybmFsMDEGA1UdHwQqMCgwJqAkoCKGIGh0dHA6Ly8xMjcuMC4wLjE6ODIw
MC92MS9wa2kvY3JsMA0GCSqGSIb3DQEBCwUAA4IBAQANNFMB3DIkkU1ZGT7KJxNk
ICcYNocNeBvfhEiJQH6EfC7eSm5vZ9Htq3Axanq20yEv7ij5TWnt5vhQ6nZ9IHbr
xZ0selef7QWxfJgE/ZrC41iUvTlvMSmgofQVNYbad064gU2ZdR6xmROV7DthM1u5
JpNK02OD0q0MKrscXcM9LG+/C2sZ7IexY5gnG1kWutJiGh1GWl3TxUjVVT2ajeOb
pyrdOzuuXsnoTQdhDi5vSLZVqHjfpEvH52Kki/St7PYmJjNeatTw27YQbNbK09Og
ROBuis60vOU8SVa8UADejk9vx+o6cShniH1inWbAp7fLSqXp78l6+5ZMoe9pKJyC
-----END CERTIFICATE-----
it would seem that the consul configuration when returning the leaf certificate to the client, doesn’t pass along the entire chain. Is this behavior working as intended? If so, is there a way for me to configure the service to pass along the entire chain?