Unable to login ACL tls: handshake message of length 92253 bytes exceeds maximum of 65536 bytes

amitoo7 · May 2, 2023, 4:53pm

[ERROR] unable to login: error="Post "https://10.118.9.252:8501/v1/acl/login": tls: handshake message of length 92253 bytes exceeds maximum of 65536 bytes"
I am getting this error from consul agent. I am new to consul service mesh.
Can anyone provide some help on this please?

I have deployed consul on k8(EKS) using helm chart and i using Vault as ConnectCA
consul version:- v1.12.0

maxb · May 2, 2023, 5:12pm

This message indicates an abnormally gigantic TLS handshake message was received, which exceeds a hardcoded limit in Go’s TLS code.

You should review the TLS configuration for this listener, and try to determine why it is sending such large handshakes.

Capturing a connection attempt with Wireshark may be useful to identify the cause.

In absence of specific evidence, my first guess would be that perhaps the server is configured to serve many huge certificates as part of its certificate chain.

amitoo7 · May 2, 2023, 5:27pm

I am using pretty standard configuration

secretsBackend:
             vault:
              enabled: true
              consulServerRole: consul-server
              consulClientRole: consul-client
              manageSystemACLsRole: consul-server-acl-init
              consulCARole: consul-ca
              connectCA:
                address: "http://vault.vault.svc.cluster.local:8200"
                rootPKIPath: connect-root/
                intermediatePKIPath: connect-intermediate-dc1/
          tls:
            enabled: true
            enableAutoEncrypt: true
            caCert:
              secretName: "pki/cert/ca"

Also In order to solve above problem i change the LeafTTL time from 3 days to 12 hours I don’t know if this is the correct solution But in some article i found this
After changing the LeafTTL to 12 hours i restarted the consul server and clients Pods

maxb · May 2, 2023, 5:35pm

Could you try running

openssl s_client -connect 10.118.9.252:8501 </dev/null

and posting the section of the output that starts with Certificate chain ?

(I’m wondering if a large number of certificates are being served there.)

If that’s not the issue, you’re going to need to do a Wireshark or tcpdump packet capture of testing a login that fails.

amitoo7 · May 8, 2023, 11:26am

Here is the output

Certificate chain
 0 s:
   i:CN = pri-18yxofs8.vault.ca.765a5740.consul
 1 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
 2 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
 3 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
 4 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
 5 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
 6 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
 7 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
 8 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
 9 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
10 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
11 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
12 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
13 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
14 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
15 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
16 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
17 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
18 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
19 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
20 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
21 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
22 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
23 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
24 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
25 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
26 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
27 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
28 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
29 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
30 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
31 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
32 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
33 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
34 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
35 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
36 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
37 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
38 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
39 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
40 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
41 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
42 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
43 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
44 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
45 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
46 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
47 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
48 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
49 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
50 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
51 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
52 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
53 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
54 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
55 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
56 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
57 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
58 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
59 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
60 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
61 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
62 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
63 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
64 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
65 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
66 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
67 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
68 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
69 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
70 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
71 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
72 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
73 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
74 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
75 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
76 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
77 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
78 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
79 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
80 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
81 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
82 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
83 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
84 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
85 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
86 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
87 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
88 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
89 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
90 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
91 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
92 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
93 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
94 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
95 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
96 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
97 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
98 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
99 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
100 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
101 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
102 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
103 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
104 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
105 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
106 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
107 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
108 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
109 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
110 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
111 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
112 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
113 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
114 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
115 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
116 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
117 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
118 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
119 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
120 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
121 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
122 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
123 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
124 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
125 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
126 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
127 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
128 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
129 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
130 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
131 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
132 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
133 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
134 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
135 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
136 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
137 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
138 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
139 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
140 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul
141 s:CN = pri-18yxofs8.vault.ca.765a5740.consul
   i:CN = pri-gpq4qmj4.vault.ca.765a5740.consul```

maxb · May 8, 2023, 11:29am

Crikey! That does not look normal at all. Why are there so many copies of either the same certificate, or certificates with the same subject-issuer pair?

That is your problem - once you get the certificate chain being served cut down to a sensible size, the issue should be resolved.

amitoo7 · May 8, 2023, 11:51am

Could you please tell me how can i cut down the certificate chain size?

maxb · May 8, 2023, 12:28pm

No, because I don’t know what it is in your environment, which has malfunctioned to create such a weird and huge certificate chain.

amitoo7 · May 8, 2023, 12:54pm

What could be the possible reasons because i am using very simple setup Helm Chart to deploy consul and using vault as CA

maxb · May 8, 2023, 8:32pm

For some reason your CA in Vault is configured to return this crazy chain.

There is no information in this conversation about how your CA in Vault is configured, so I have no ability to guess why it is configured that way.

blake · May 12, 2023, 10:45pm

Hi @amitoo7,

It seems like you might be running into this bug.

github.com/hashicorp/consul

Prune expired intermediate CA certs from stored roots

opened 07:15PM - 11 Aug 22 UTC

closed 10:34PM - 02 Sep 22 UTC

kyhavlov

theme/connect theme/certificates

Consul currently doesn't remove expired intermediates from the stored root CA st…ruct - it only ever appends new ones: - https://github.com/hashicorp/consul/blob/201d1458c307e9b04fe3d6f22a2ad9d6d1358bdd/agent/consul/leader_connect_ca.go#L538 - https://github.com/hashicorp/consul/blob/201d1458c307e9b04fe3d6f22a2ad9d6d1358bdd/agent/consul/leader_connect_ca.go#L1101 This is currently only handled when a root rotation happens, because the old list of intermediates will get left behind, but when the intermediate cert TTL is significantly lower than the root's, it's possible to have many intermediate certs build up in the list. This can cause long certificate chains to be sent to Envoy proxies, which in turn can cause the TLS handshake to fail due to too many certs in the chain. We should add some logic to prune old expired intermediate certs from the list when a new one is appended, to prevent them from building up indefinitely.

This issue was fixed in Consul 1.12.5 by PR #14429.

connect: Fixed an issue where intermediate certificates could build up in the root CA because they were never being pruned after expiring. [GH-14429]

I recommend upgrading from 1.12.0 to 1.12.5 to resolve this issue.

amitoo7 · May 15, 2023, 3:17pm

Thank @blake for your help. It did solve the issue.