Consul version: v1.9.2
Nomad version: v1.0.2
An ingress job is running in nomad with envoy sidecar proxy using consul connect. Consul is running on both http/8500 and https/8501 (for backward compatibility) but has rpc configs set to false. Nomad and consul both log warnings below:
Nomad ingress job logs -
2021-02-02T21:48:59.057Z [WARN] client.alloc_runner.runner_hook: error proxying from Consul: alloc_id=2373fdf7-189e-31df-0e82-b00b3655b391 error="read tcp 127.0.0.1:48020->127.0.0.1:8502: read: connection reset by peer" dest=127.0.0.1:8502 src_local=/opt/nomad/alloc/2373fdf7-189e-31df-0e82-b00b3655b391/alloc/tmp/consul_grpc.sock src_remote=@ bytes=0
Consul logs for this port:
2021-02-02T21:48:59.057Z [WARN] agent: grpc: Server.Serve failed to complete security handshake from "127.0.0.1:48020": tls: first record does not look like a TLS handshake.
We also observed that when https is enabled, rpc port(8502) starts running with tls too.
Ran "openssl s_client -connect NODE_IP:8502" command and got following response -
- https enabled, RPC configs (verify_incoming & verify_outgoing) are set to false
depth=0 CN = server.XXXX.consul
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = server.XXXX.consul
verify error:num=21:unable to verify the first certificate
verify return:1Certificate chain
0 s:/CN=server.XXXX.consul
i:/C=US/ST=CA/L=San Francisco/street=101 Second Street/postalCode=94105/O=HashiCorp Inc./CN=Consul Agent CA 76243903559371484871955553276941343535Server certificate
BEGIN CERTIFICATE
MIIC2TCCAn+gAwIBAgIRALqE9myBIai3kEoWwhxFZ8EwCgYIKoZIzj0EAwIwgbgx
CzAJBgNVBAYTAlVTMQswCQYDVQQIEwJDQTEWMBQGA1UEBxMNU2FuIEZyYW5jaXNj
bzEaMBgGA1UECRMRMTAxIFNlY29uZCBTdHJlZXQxDjAMBgNVBBETBTk0MTA1MRcw
FQYDVQQKEw5IYXNoaUNvcnAgSW5jLjE/MD0GA1UEAxM2Q29uc3VsIEFnZW50IENB
IDc2MjQzOTAzNTU5MzcxNDg0ODcxOTU1NTUzMjc2OTQxMzQzNTM1MB4XDTIxMDIw
MjEzNTUyMlGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAwKQYDVR0OBCIEIOnJ
hYCvHS3ujBiE1HrkiXsEt7DCIcg9M4kz5jVyoVSpMCsGA1UdIwQkMCKAIAZBod1y
rec3Fl9IzTY/CBMMyyA6006svfyDHD0SDsLEMFoGA1UdEQRTMFGCFWNvbnN1bC5z
ZXJ2aWNlLmNvbnXXXX4bWXUG2tACmQExHAiBeI2cwKFJcjm7RhQMqeblz7itB
jSNCoUB8TvDYFIteRA==
END CERTIFICATE
subject=/CN=server.XXXX.consul
issuer=/C=US/ST=CA/L=San Francisco/street=101 Second Street/postalCode=94105/O=HashiCorp Inc./CN=Consul Agent CA 76243903559371484871955553276941343535No client certificate CA names sent
Server Temp Key: ECDH, X25519, 253 bitsSSL handshake has read 1136 bytes and written 289 bytes
New, TLSv1/SSLv3, Cipher is ECDHE-ECDSA-AES256-GCM-SHA384
Server public key is 256 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-ECDSA-AES256-GCM-SHA384
Session-ID: D88E00285E9A651E67BBC731D814AC0FA26FED3CC019884402AD182156502FF7
Session-ID-ctx:
Master-Key: 92D368F3D240D574A0FB1B7BF32D25FD12154BC832D3BAFBDA7ADB070E47DA8CB79F6A53BFEC195818A6C2938A9493E7
TLS session ticket:
0000 - 4d 92 a5 2f 17 02 1a 85-2f f2 eb e9 ab 1e f7 27 M…/…/…’
0010 - d8 34 64 35 a9 47 70 d3-d0 7f ec e8 51 b0 dc 1b .4d5.Gp…Q…
0020 - 3c c9 ba 28 5d ae fd 40-8a ce cb 36 5b 9e a4 39 <…(]…@…6[…9
0030 - 60 71 e0 96 87 e2 7b a9-21 ae b6 31 85 dc e3 19 `q…{.!..1…
0040 - 2e 6a ca 10 b1 66 83 81-7f 11 8b f4 22 04 22 c1 .j…f…“.”.
0050 - 2c 7f 63 5f 67 6f 1f 03-80 9c 36 cd 62 ae 92 e2 ,.c_go…6.b…
0060 - 0e f3 42 57 39 ab 40 8b-8c d3 aa 7e 73 3f 89 0e …BW9.@…~s?..
0070 - cd eb 78 6c ef 55 8a 06-80 4e 56 10 1e 97 1f c1 …xl.U…NV…
0080 - a9 .Start Time: 1612305064 Timeout : 7200 (sec) Verify return code: 21 (unable to verify the first certificate)
- https disabled
l-47.140.1/libressl-2.8/ssl/ssl_pkt.c:386:
no peer certificate available
No client certificate CA names sent
SSL handshake has read 5 bytes and written 0 bytes
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : 0000
Session-ID:
Session-ID-ctx:
Master-Key:
Start Time: 1612303506
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Two questions:
- RPC port running on tls when https config is set in consul - is this behavior expected?
- What config needs to be added in nomad for it to be able to talk to consul connect when RPC port (8502) has tls enabled? From the documentation here(Secure Nomad Jobs with Consul Service Mesh | Nomad | HashiCorp Developer) it doesn’t seem like extra configurations are needed.