Clients not sending TLS Certs

Consul
1

Hello, I’ve been stuck with a TLS Client cert issues for a while…I deployed using Helm… I created the certs with the built-in CA described here: https://learn.hashicorp.com/tutorials/consul/tls-encryption-secure

These are the errors from the server logs:

[ERROR] agent.server.rpc: TLS handshake failed: conn=from=13.xx.xx.xxx:38462 error=“tls: client didn’t provide a certificate”

These are from the clients:

[ERROR] agent.client.memberlist.lan: memberlist: Received invalid msgType (22) from=13.xx.xx.xxx:59562

[WARN] agent: grpc: Server.Serve failed to create ServerTransport: connection error: desc = “transport: http2Server.HandleStreams received bogus greeting from client: “GET /metrics HTTP/1.1\r\nH””

On the clients the tls.crt file does exist at: /consul/tls/ca

I’ve tried several configurations and so far the errors persist. This is my current config file that I use with helm:

global:
enabled: true
name: consul
datacenter: dc1
image: ‘consul:1.8.4’
imageEnvoy: envoyproxy/envoy-alpine:v1.14.2
serverAdditionalDNSSANs: ["‘consul.service.consul’"]

tls:
enabled: true
enableAutoEncrypt: true
httpsOnly: true
verify: false
caCert:
secretName: consul-ca-cert
secretKey: tls.crt
caKey:
secretName: consul-ca-key
secretKey: tls.key

lifecycleSidecarContainer:
resources:
requests:
memory: ‘20Mi’
cpu: ‘20m’
limits:
memory: ‘50Mi’
cpu: ‘20m’

dns:
enabled: true

server:
enabled: true
replicas: 3
bootstrapExpect: 3
storage: 10Gi
connect: true
server.extraConfig: |
{
“allow_tls”: {
“enabled”: true
“enable_agent_tls_for_checks”: true
}
}

sidecarProxy:
resources:
requests:
memory: 100Mi
cpu: 100m
limits:
memory: 100mi
cpu: 100m

connectInject:
enabled: true
default: false

services:
backend: backend.service.consul

ui:
enabled: true
service:
type: ‘LoadBalancer’

client:
enabled: true
grpc: true
exposeGossipPorts: true
extraConfig: |
{
“auto_encrypt”: {
“tls”: true
}
}

syncCatalog:
enabled: true
default: true
toConsul: true
toK8s: true

affinity: null

resources:
requests:
memory: ‘25Mi’
cpu: ‘20m’
limits:
memory: ‘50Mi’
cpu: ‘20m’

thank you…

2

Hi! I haven’t used Helm charts, but it looks like you’re just including your CA cert and key. You need a separate cert and separate key for your client(s).

It looks like you aren’t verifying them, which is fine. If you were, all those certs would need to be signed by a common/recognised CA.

But I think your first step is to provide client-specific TLS configuration.

3

Thanks for the reply, I’ve generated the client certs, but unfortunately the docs are not clear on how to add this to the helm chart configs, so I guess I’ll just have to try to add it to extraConfigs for the client section, maybe the same way the certs have been added for the servers in the Global section…

4

I added the client certs and mounted it as a extra volume and it seems to be accessing the secrets, but it doesn’t like the certificate even though I generated it using:

The update config:
client:
enabled: true
grpc: true
exposeGossipPorts: true
extraConfig: |
{
“auto_encrypt”: {
“tls”: true
}
}
extraVolumes:

  • type: “secret”
    name: “consul-ca-client-cert”
    load: false
  • type: “secret”
    name: “consul-ca-client-key-cert”
    load: false

The errors:

agent: yamux: Failed to read header: remote error: tls: bad certificate
[ERROR] agent.client.memberlist.lan: memberlist: Received invalid msgType (22)

5

“Bad certificate” is normally a trust issue, I believe. Did you create the server and client certificates with the same Consul CA? And is the common name (CN) of both the same datacenter?

Can you include details of both with openssl x509 -in <your cert> -noout -text, please?

6

Sorry for the slow reply…

I created the certs per the instructions here: https://learn.hashicorp.com/tutorials/consul/tls-encryption-secure
Also, the name of the datacenter is dc1

Here is the output of client.dc1.consul.crt:

Certificate:
Data:
Version: 1 (0x0)
Serial Number:
34:e4:e9:6f:0f:10:91:f4:01:87:17:98:94:ba:4f:61:20:70:41:c0
Signature Algorithm: ecdsa-with-SHA256
Issuer: C = US, ST = CA, L = San Francisco, street = 101 Second Street, postalCode = 94105, O = HashiCorp Inc., CN = Consul Agent CA 61640881572597635504608575816850140554
Validity
Not Before: Nov 17 22:17:37 2020 GMT
Not After : Dec 17 22:17:37 2020 GMT
Subject: CN = client.dc1.consul
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:d1:86:a8:1e:09:e1:2c:46:c3:ca:fd:7c:7f:22:
f0:9e:af:01:e6:ba:13:b4:4f:e9:56:b2:3b:d2:f3:
a9:48:61:22:1a:f0:d9:62:dc:d5:98:96:7f:fe:c0:
c9:27:38:01:84:df:0d:33:51:d8:66:70:fb:60:32:
db:4f:61:13:54:fc:56:d1:8d:15:a3:e0:46:df:27:
b7:70:d4:5e:a3:5a:57:3f:3e:d8:b3:8a:01:5a:0c:
af:95:60:e4:55:0d:28:6d:15:04:e8:22:73:7d:58:
29:a3:fb:69:3c:a4:55:f4:fe:e6:18:6a:79:84:fb:
3b:48:03:ad:17:b7:04:1e:0a:d6:5f:62:dc:66:8b:
25:9f:c3:6f:7c:48:6e:65:71:a9:b4:58:58:09:f4:
97:b5:77:72:6e:62:0c:d0:1f:8e:38:51:1b:c4:a2:
6f:9e:e3:2f:27:ff:fc:39:f4:3c:c7:81:3d:d4:e9:
45:12:e5:3e:3e:79:07:bc:d2:e0:89:0a:3c:3e:99:
b3:eb:00:83:c9:68:43:da:58:40:79:6d:a7:ef:a5:
fd:1b:a9:01:73:dc:e1:aa:4f:17:b0:ea:68:71:eb:
50:5f:ed:4b:3b:f2:7e:23:08:4f:62:e0:7c:a1:ec:
27:f8:71:c8:07:37:32:e6:38:eb:a0:8c:ed:c9:f4:
48:51
Exponent: 65537 (0x10001)
Signature Algorithm: ecdsa-with-SHA256
30:44:02:20:12:2b:77:24:b6:8b:2a:56:1f:bb:70:82:4a:b3:
c5:0d:f6:2a:72:f4:f8:ab:d3:2b:ca:dc:49:59:f8:43:c8:bb:
02:20:4f:8d:e1:8a:59:4d:9f:32:5a:e7:4c:6d:38:76:53:98:
0d:60:a8:60:c1:10:4e:af:5e:c2:51:82:61:ee:7b:bb

Here is the output of server1.dc1.consul.crt:

Certificate:
Data:
Version: 1 (0x0)
Serial Number:
34:e4:e9:6f:0f:10:91:f4:01:87:17:98:94:ba:4f:61:20:70:41:bf
Signature Algorithm: ecdsa-with-SHA256
Issuer: C = US, ST = CA, L = San Francisco, street = 101 Second Street, postalCode = 94105, O = HashiCorp Inc., CN = Consul Agent CA 61640881572597635504608575816850140554
Validity
Not Before: Nov 17 22:13:17 2020 GMT
Not After : Dec 17 22:13:17 2020 GMT
Subject: CN = server.dc1.consul
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:d4:40:3e:1f:1c:57:a3:ac:45:48:fe:46:1b:d0:
66:eb:07:9c:ba:ae:d3:de:b5:74:5f:b9:7a:47:4e:
57:c9:99:f5:a2:6f:22:a2:6d:e5:3e:1d:ae:48:26:
8e:ec:99:97:3e:15:02:06:b0:1f:12:b6:1d:cd:b9:
16:ea:0a:b7:82:b8:5c:54:f6:b8:99:ad:b7:e3:dc:
8e:3b:8c:71:3c:06:a1:08:09:e0:a9:21:23:39:4f:
de:17:d9:b1:65:3c:c5:bf:b0:08:13:68:97:0b:14:
17:94:a3:c8:fc:b4:6a:fd:95:16:2b:a3:cb:a7:ad:
98:8e:c2:b6:9b:48:93:3d:0a:4f:18:d9:af:06:ea:
21:0f:ee:d7:5b:f7:17:09:c6:f8:19:64:7d:d8:25:
4f:2d:6f:89:91:f4:61:da:f1:e8:92:60:38:52:33:
86:64:81:17:f5:52:99:a7:07:d8:19:8a:a3:32:57:
a1:85:94:b1:63:99:44:4e:ae:bc:80:b5:e6:3e:db:
ef:d2:bd:2f:1d:5b:fb:56:3b:12:91:f6:2d:d8:5b:
7f:82:95:de:32:24:0a:6b:c6:3f:91:14:bf:6b:49:
a4:08:21:ec:3d:d4:63:4a:a5:a1:62:79:69:16:b6:
ec:ee:05:56:36:f2:2b:90:a7:dc:8f:f7:d1:f7:c7:
d6:a9
Exponent: 65537 (0x10001)
Signature Algorithm: ecdsa-with-SHA256
30:44:02:20:49:3a:69:a7:27:37:0b:4c:a4:57:1b:76:d0:db:
ee:43:d6:db:ad:e0:98:5c:9b:df:d0:5b:a6:d2:e0:02:98:6f:
02:20:05:94:f0:2a:78:a3:59:84:67:2a:b6:c4:09:54:b8:8d:
2e:72:c3:cf:2e:b5:2a:dc:c7:30:69:b7:a7:cc:e0:fa

7

After examining the logs in the lead server, it looks like my recent updates are working. But I still see only one error, that points to one IP:

“TLS handshake failed”

The error points to a single IP, I traced that IP to a prometheus kubecost instance that should have nothing to do with Consul. Any ideas why this is occurring?

8

\o/ :grinning:

H’m, again, this is really outside my bailiwick, but, I’d imagine, for Kubecost to do a decent job of cost estimation, it’d have to get pretty nosy. :slight_smile: Either way, that error is likely because the instance isn’t presenting a certificate signed by your CA.

9

Thanks for your help, much appreciated…

1 Like