Nomad mesh integration fails when talking to Consul with both https and http enabled

Consul version: v1.9.2
Nomad version: v1.0.2

An ingress job is running in nomad with envoy sidecar proxy using consul connect. Consul is running on both http/8500 and https/8501 (for backward compatibility) but has rpc configs set to false. Nomad and consul both log warnings below:

Nomad ingress job logs -
2021-02-02T21:48:59.057Z [WARN] client.alloc_runner.runner_hook: error proxying from Consul: alloc_id=2373fdf7-189e-31df-0e82-b00b3655b391 error="read tcp> read: connection reset by peer" dest= src_local=/opt/nomad/alloc/2373fdf7-189e-31df-0e82-b00b3655b391/alloc/tmp/consul_grpc.sock src_remote=@ bytes=0

Consul logs for this port:
2021-02-02T21:48:59.057Z [WARN] agent: grpc: Server.Serve failed to complete security handshake from "": tls: first record does not look like a TLS handshake.

We also observed that when https is enabled, rpc port(8502) starts running with tls too.

Ran "openssl s_client -connect NODE_IP:8502" command and got following response -

  1. https enabled, RPC configs (verify_incoming & verify_outgoing) are set to false

depth=0 CN = server.XXXX.consul
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = server.XXXX.consul
verify error:num=21:unable to verify the first certificate
verify return:1

Certificate chain
0 s:/CN=server.XXXX.consul
i:/C=US/ST=CA/L=San Francisco/street=101 Second Street/postalCode=94105/O=HashiCorp Inc./CN=Consul Agent CA 76243903559371484871955553276941343535

Server certificate
issuer=/C=US/ST=CA/L=San Francisco/street=101 Second Street/postalCode=94105/O=HashiCorp Inc./CN=Consul Agent CA 76243903559371484871955553276941343535

No client certificate CA names sent
Server Temp Key: ECDH, X25519, 253 bits

SSL handshake has read 1136 bytes and written 289 bytes

New, TLSv1/SSLv3, Cipher is ECDHE-ECDSA-AES256-GCM-SHA384
Server public key is 256 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Protocol : TLSv1.2
Session-ID: D88E00285E9A651E67BBC731D814AC0FA26FED3CC019884402AD182156502FF7
Master-Key: 92D368F3D240D574A0FB1B7BF32D25FD12154BC832D3BAFBDA7ADB070E47DA8CB79F6A53BFEC195818A6C2938A9493E7
TLS session ticket:
0000 - 4d 92 a5 2f 17 02 1a 85-2f f2 eb e9 ab 1e f7 27 M…/…/…’
0010 - d8 34 64 35 a9 47 70 d3-d0 7f ec e8 51 b0 dc 1b .4d5.Gp…Q…
0020 - 3c c9 ba 28 5d ae fd 40-8a ce cb 36 5b 9e a4 39 <…(]…@…6[…9
0030 - 60 71 e0 96 87 e2 7b a9-21 ae b6 31 85 dc e3 19 `q…{.!..1…
0040 - 2e 6a ca 10 b1 66 83 81-7f 11 8b f4 22 04 22 c1 .j…f…".".
0050 - 2c 7f 63 5f 67 6f 1f 03-80 9c 36 cd 62 ae 92 e2 ,.c_go…6.b…
0060 - 0e f3 42 57 39 ab 40 8b-8c d3 aa 7e 73 3f 89 0e …BW9.@…~s?..
0070 - cd eb 78 6c ef 55 8a 06-80 4e 56 10 1e 97 1f c1 …xl.U…NV…
0080 - a9 .

Start Time: 1612305064
Timeout   : 7200 (sec)
Verify return code: 21 (unable to verify the first certificate)

  1. https disabled


no peer certificate available

No client certificate CA names sent

SSL handshake has read 5 bytes and written 0 bytes

New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Protocol : TLSv1.2
Cipher : 0000
Start Time: 1612303506
Timeout : 7200 (sec)
Verify return code: 0 (ok)

Two questions:

  1. RPC port running on tls when https config is set in consul - is this behavior expected?
  2. What config needs to be added in nomad for it to be able to talk to consul connect when RPC port (8502) has tls enabled? From the documentation here(Secure Nomad Jobs with Consul Service Mesh | Nomad - HashiCorp Learn) it doesn’t seem like extra configurations are needed.

Missed mentioning this -
Nomad(v 1.0.2) mesh integration is enabled and has an ingress job running. Nomad is still communicating with consul over http and 8500 (consul stanza config below)

consul {
address = “”
token = “XXXX”
ssl = false