TLS client didn't provide a certificate problem

Consul
1

Hi all

Using Consul v1.12.2 and have also installed Envoy 1.21.1 installed successfully.

When we run the following command:

consul connect envoy -sidecar-for frontend-1 -- --log-level debug

we see the following output:

[2022-06-11 12:29:24.315][25426][debug][connection] [source/common/network/connection_impl.cc:896] [C6] connecting to 127.0.0.1:8502
[2022-06-11 12:29:24.315][25426][debug][connection] [source/common/network/connection_impl.cc:916] [C6] connection in progress
[2022-06-11 12:29:24.316][25426][debug][connection] [source/common/network/connection_impl.cc:680] [C6] connected
[2022-06-11 12:29:24.316][25426][debug][connection] [source/common/network/connection_impl.cc:936] [C6] connected on local interface 'lo'
[2022-06-11 12:29:24.317][25426][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:225] [C6] TLS error: 268436498:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICATE
[2022-06-11 12:29:24.317][25426][debug][connection] [source/common/network/connection_impl.cc:249] [C6] closing socket: 0
[2022-06-11 12:29:24.317][25426][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:225] [C6] TLS error: 268436498:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICATE
[2022-06-11 12:29:24.317][25426][debug][client] [source/common/http/codec_client.cc:110] [C6] disconnect. resetting 0 pending requests
[2022-06-11 12:29:24.317][25426][debug][pool] [source/common/conn_pool/conn_pool_base.cc:443] [C6] client disconnected, failure reason: TLS error: 268436498:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICATE
[2022-06-11 12:29:24.317][25426][debug][router] [source/common/router/router.cc:1156] [C0][S8877616002743877539] upstream reset: reset reason: connection failure, transport failure reason: TLS error: 268436498:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICATE
[2022-06-11 12:29:24.317][25426][debug][http] [source/common/http/async_client_impl.cc:101] async http request response headers (end_stream=true):
':status', '200'
'content-type', 'application/grpc'
'grpc-status', '14'
'grpc-message', 'upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268436498:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICATE'

[2022-06-11 12:29:24.317][25426][debug][config] [./source/common/config/grpc_stream.h:202] DeltaAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268436498:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICATE
[2022-06-11 12:29:24.317][25426][debug][config] [source/common/config/grpc_subscription_impl.cc:113] gRPC update for type.googleapis.com/envoy.config.cluster.v3.Cluster failed
[2022-06-11 12:29:24.317][25426][debug][config] [source/common/config/grpc_subscription_impl.cc:113] gRPC update for type.googleapis.com/envoy.config.listener.v3.Listener failed
[2022-06-11 12:29:24.317][25426][debug][pool] [source/common/conn_pool/conn_pool_base.cc:410] invoking idle callbacks - is_draining_for_deletion_=false
[2022-06-11 12:29:27.543][25426][debug][main] [source/server/server.cc:242] flushing stats
[2022-06-11 12:29:27.543][25426][debug][main] [source/server/server.cc:252] Envoy is not fully initialized, skipping histogram merge and flushing stats

It appears that there is a TLS problem, but the all of the environment variables are set as follows:

CONSUL_HTTP_ADDR=https://127.0.0.1:8501
CONSUL_HTTP_TOKEN=xxxxxxx-xxxx-xxx
CONSUL_GRPC_ADDR=https://127.0.0.1:8502
CONSUL_CACERT=/etc/consul.d/certs/consul-agent-ca.pem
CONSUL_HTTP_SSL=true
CONSUL_CLIENT_CERT=/etc/consul.d/certs/dc1-client-consul-0.pem
CONSUL_CLIENT_KEY=/etc/consul.d/certs/dc1-client-consul-0-key.pem

The consul agent log file also shows the following:

Jun 11 12:30:53 ip-10-0-3-98.ap-southeast-2.compute.internal consul[15603]: 2022-06-11T12:30:53.126Z [WARN]  agent: Check is now critical: check=service:frontend-1-sidecar-proxy
Jun 11 12:31:01 ip-10-0-3-98.ap-southeast-2.compute.internal consul[15603]: 2022-06-11T12:31:01.477Z [WARN]  agent: [core]grpc: Server.Serve failed to complete security handshake from "127.0.0.1:52324": tls: client didn't provide a certificate
Jun 11 12:31:03 ip-10-0-3-98.ap-southeast-2.compute.internal consul[15603]: 2022-06-11T12:31:03.127Z [WARN]  agent: Check socket connection failed: check=service:frontend-1-sidecar-proxy error="dial tcp 10.0.3.98:20000: connect: connection refused"
Jun 11 12:31:03 ip-10-0-3-98.ap-southeast-2.compute.internal consul[15603]: 2022-06-11T12:31:03.127Z [WARN]  agent: Check is now critical: check=service:frontend-1-sidecar-proxy

Hoping someone might be able to shed some light on what the problem is.

Thank you in advance.

2

I don’t know anything about envoy but this sounds like either a network routing problem or that the port isn’t bound.

Use netstat on the destination server to see if port 20000 is bound locally:

$ netstat -an | grep 20000

If it shows up as bound then you can check the connectivity between the two, from the connecting host:

$ ping 10.0.398 
$ curl 10.0.3.98:20000

The output (which maybe an error) will tell you if the two instances can reach out via TCP … from there it’s application configuration issue.

3

@James78 I just ran into the same issue with a Nomad envoy sidecar.

This seems to be the problem: `verify_outgoing` not supported on gRPC listener; `verify_incoming` breaks Consul Connect envoy sidecars · Issue #13088 · hashicorp/consul · GitHub

This problem was introduced with 1.12.x when setting tls.defaults.verify_incoming = true.
The current “workaround” (return to pre 1.12.x behaviour), is to set tls.grpc.verify_incoming = false.

1 Like