Exposed Services are not accessible after Consul Upgrade 1.10.4

Hi,

Recently my Team had to upgrade Consul to version 1.10.4 using Helm Chart deployment.
After upgrade we noticed that service discovery of Kubernetes services with exposed port stopped working.

We’re mapping our services using Ambassador mappings.

Type of service mappings which works fine:
<service-name>-sidecar-proxy

Type of service mappings which are failing for us:
http://<name-of-service>.<namespace>:<exposed-service-port>

Local port-forwarding of services shows that services working fine.
kubectl port-forward -n <namespace> svc/<name-of-service> <exposed-service-port>:<exposed-service-port>

When we try to access mapping of service we’re getting following error response from Consul:

upstream connect error or disconnect/reset before headers. reset reason: connection termination

After checking in Ambassador pod logs we see following errors:

[2022-01-13 11:56:03.452][54][debug][router] [source/common/router/router.cc:426] [C396806][S3367325754288988140] cluster 'cluster_dev_service_<service-name>_dev_8081_dev' match for URL '/api/<service-name>/health'
[2022-01-13 11:56:03.452][54][debug][router] [source/common/router/router.cc:583] [C396806][S3367325754288988140] router decoding headers:
':method', 'GET'
':authority', '<hostname-mapped-with-ambassador-load-balancer>'
':scheme', 'http'
':path', '/health'
'cache-control', 'max-age=0'
'sec-ch-ua', '" Not A;Brand";v="99", "Chromium";v="96", "Google Chrome";v="96"'
'sec-ch-ua-mobile', '?0'
'sec-ch-ua-platform', '"Windows"'
'upgrade-insecure-requests', '1'
'user-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'
'accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'
'sec-fetch-site', 'none'
'sec-fetch-mode', 'navigate'
'sec-fetch-user', '?1'
'sec-fetch-dest', 'document'
'accept-encoding', 'gzip, deflate, br'
'accept-language', 'en-US,en;q=0.9'
'cookie', '_ga=GA1.2.1655039019.1626686851; _gid=GA1.2.843066054.1641801462; PF=xxx'
'x-forwarded-for', '10.145.202.99'
'x-forwarded-proto', 'https'
'x-envoy-internal', 'true'
'x-request-id', 'xxx'
'x-envoy-expected-rq-timeout-ms', '10000000'
'x-b3-traceid', 'xxx'
'x-b3-spanid', 'xxx'
'x-b3-sampled', '1'
'x-envoy-original-path', '/api/<service-name>/health'

[2022-01-13 11:56:03.452][54][debug][pool] [source/common/http/conn_pool_base.cc:71] queueing request due to no available connections
[2022-01-13 11:56:03.452][54][debug][pool] [source/common/conn_pool/conn_pool_base.cc:53] creating a new connection
[2022-01-13 11:56:03.452][54][debug][client] [source/common/http/codec_client.cc:35] [C399035] connecting
[2022-01-13 11:56:03.452][54][debug][connection] [source/common/network/connection_impl.cc:753] [C399035] connecting to <IP_ADDRESS_OF_EXPOSED_SERVICE>:8081
[2022-01-13 11:56:03.452][54][debug][connection] [source/common/network/connection_impl.cc:769] [C399035] connection in progress
[2022-01-13 11:56:03.452][54][debug][http2] [source/common/http/http2/codec_impl.cc:862] [C396807] stream closed: 0
[2022-01-13 11:56:03.452][54][debug][connection] [source/common/network/connection_impl.cc:616] [C399035] connected
[2022-01-13 11:56:03.452][54][debug][client] [source/common/http/codec_client.cc:73] [C399035] connected
[2022-01-13 11:56:03.452][54][debug][pool] [source/common/conn_pool/conn_pool_base.cc:146] [C399035] attaching to next request
[2022-01-13 11:56:03.452][54][debug][pool] [source/common/conn_pool/conn_pool_base.cc:73] [C399035] creating stream
[2022-01-13 11:56:03.452][54][debug][router] [source/common/router/upstream_request.cc:342] [C396806][S3367325754288988140] pool ready
[2022-01-13 11:56:03.453][54][debug][connection] [source/common/network/connection_impl.cc:584] [C399035] remote close
[2022-01-13 11:56:03.453][54][debug][connection] [source/common/network/connection_impl.cc:208] [C399035] closing socket: 0
[2022-01-13 11:56:03.453][54][debug][client] [source/common/http/codec_client.cc:92] [C399035] disconnect. resetting 1 pending requests
[2022-01-13 11:56:03.453][54][debug][client] [source/common/http/codec_client.cc:115] [C399035] request reset
[2022-01-13 11:56:03.453][54][debug][router] [source/common/router/router.cc:1022] [C396806][S3367325754288988140] upstream reset: reset reason connection termination
[2022-01-13 11:56:03.453][54][debug][http] [source/common/http/conn_manager_impl.cc:1520] [C396806][S3367325754288988140] Sending local reply with details upstream_reset_before_response_started{connection termination}
[2022-01-13 11:56:03.453][54][debug][http] [source/common/http/conn_manager_impl.cc:1777] [C396806][S3367325754288988140] encoding headers via codec (end_stream=false):
':status', '503'
'content-length', '95'
'content-type', 'text/plain'
'date', 'Thu, 13 Jan 2022 11:56:03 GMT'
'server', 'envoy'

Example of Ambassador mapping which we’re using in regards to this issue:

apiVersion: getambassador.io/v2
kind: Mapping
metadata:
  creationTimestamp: "2022-01-04T13:10:27Z"
  generation: 6
  name: <service-name>-mapping
  namespace: dev
  resourceVersion: "5300550"
  selfLink: /apis/getambassador.io/v2/namespaces/dev/mappings/<service-name>-mapping
  uid: a9a3627f-ab0a-47fa-8662-53dad2311516
spec:
  connect_timeout_ms: 10000000
  host: <hostname-mapped-with-ambassador-load-balancer>
  idle_timeout_ms: 10000000
  prefix: /api/<service-name>/
  retry_policy:
    num_retries: 10
    retry_on: 5xx
  rewrite: /
  service: http://<service-name>.<namespace>:<exposed-service-port>
  timeout_ms: 10000000

As for Consul upgrade we added two CRD as per Consul documentation specification:
proxy-defaults ← we’re setting here default “http” protocol
service-defaults ← created for services which use “grpc” protocol

Previous Consul Version: 1.8.2
Previous Consul Helm Version: 0.24.1
Current Consul Version: 1.10.4
Current Consul Helm Version: 0.38.0
Ambassador Version: 1.11.2
Ambassador Helm Version: 6.5.20
Ambassador Consul Connect: ambassador_pro:consul_connect_integration-0.11.0
K8S Version: 1.19

Can anyone help us out or guide what part of configuration we may missed after doing Consul upgrade?