No issuer SSL error in Consul client secondary DC

Hi everyone,

This is a high-level overview of our current architecture:

  • Consul client agents in AWS EKS (daemonset)
  • Consul servers in AWS ECS Fargate (5 servers)
  • We have our primary DCs in one region for all environments

Consul client version: 1.11.1
Consul server version: 1.11.1

We have auto_encrypt enabled for our Consul cluster, and we generate our own server certificates. We ran into an issue whereby in the primary DC, the client certificates work fine when we try to pass a request from a local Kubernetes service to the client.

But when trying to issue an HTTPS request from a pod in the secondary DC to a Consul client agent in the secondary DC, we face an SSL error (60) X509-issuer not found. When we inspect the certificate chain of the Consul client in the secondary DC, we find the “server” config to have an empty issuer.

For reference, we supplied the Consul client in the primary DC with the Consul Connect root CA that was fetched from the primary DC servers (using curl -s -k {IP}:{PORT}/v1/connect/ca/roots). And we were able to make requests such as curl --cacert client-cert.pem -H "X-Consul-Token":"XX" https://localhost:8501/v1/catalog/services after.

The same doesn’t work in a secondary DC. We would fetch the certificate from the secondary DC, but would get two: a root and a intermediate certificate (both with a valid issuer). But when we try to make an HTTPS request with any of these certs, we would get the error above.

Exact error:

curl --cacert <client intermediate cert> -H "X-Consul-Token":"XX" https://localhost:8501/v1/catalog/services
curl: (60) SSL: couldn't get X509-issuer name!
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Client cert (secondary DC) – no issuer

openssl s_client -connect localhost:8501

CONNECTED(00000003)
Can't use SSL_get_servername
depth=1
verify error:num=30:authority and subject key identifier mismatch
verify return:1
depth=0
verify return:1
---
Certificate chain
 0 s:
   i: 
   a: [REDACTED]
   v: [REDACTED]
 1 s:
   i:CN = [REDACTED]
   a:PKEY: [REDACTED]
   v:NotBefore: [REDACTED]
 2 s:
   i:CN = [REDACTED]
   a:PKEY: [REDACTED]
   v:NotBefore: [REDACTED]
---
Server certificate
[REDACTED]
subject=
issuer=
---
No client certificate CA names sent
Peer signing digest: [REDACTED]
Peer signature type: [REDACTED]
Server Temp Key: [REDACTED]

My question is as follows:

  1. What could be some possible reasons for the Consul client to have an “issuer” in the primary DC but not the secondary?
  2. Does auto_encrypt help create certificates for clients in both the the primary and secondary DCs?