How to add envoy escape hatch config for external services using terminating gateways

Consul
1

I have a working mesh setup with tproxy enabled. I’ve connected external RDS using a terminating gateway. Everything works fine at a normal load but under moderate load, I am seeing a lot of upstream connection failures in the terminating gateway i.e. a lot of failures when the envoy is trying to connect to RDS. However, I do not see any connection failures if I put the same load without mesh.

I think I would see better results if I could reuse TCP connections. To do that, I need to update the envoy config. I’ve added ethe scape hatch config as shown below but it’s not getting reflected when I hit /clusters of envoy. Here are my configs

  1. RDS service was registered using http://localhost:8500/v1/catalog/register
{
  "Node": "rds_reader_node",
  "Address": "<rds DNS here>",
  "NodeMeta": {
    "external-node": "true",
    "external-probe": "true"
  },
  "Service": {
    "ID": "rds_reader",
    "Service": "rds-reader",
    "Port": 3306
  }
}
  1. Terminating gateway config entry
Kind = "terminating-gateway"
Name = "terminating-gateway"

Services = [
  {
    Name = "rds-reader"
  }
]

  1. Terminating gateway proxy is started using consul connect envoy -gateway=terminating -register -service terminating-gateway

  2. terminating-gateway envoy config to increase TCP keep alive and other settings

service {
  name = "terminating-gateway"
  kind = "terminating-gateway"
  port = 8443

  proxy = {
    config = {
      envoy_static_resources = <<EOF
          {
            "static_resources": {
              "clusters": [
                {
                  "name": "rds-reader",
                  "connect_timeout": "0.25s",
                  "type": "LOGICAL_DNS",
                  "lb_policy": "ROUND_ROBIN",
                  "load_assignment": {
                    "cluster_name": "rds-reader",
                    "endpoints": [
                      {
                        "lb_endpoints": [
                          {
                            "endpoint": {
                              "address": {
                                "socket_address": {
                                  "address": "<RDS DNS HERE>",
                                  "port_value": 3306
                                }
                              }
                            }
                          }
                        ]
                      }
                    ]
                  },
                  "circuit_breakers": {
                    "thresholds": [
                      {
                        "priority": "DEFAULT",
                        "max_connections": 1000,
                        "max_pending_requests": 1000,
                        "max_requests": 1000
                      }
                    ]
                  },
                  "upstream_connection_options": {
                    "tcp_keepalive": {
                      "keepalive_time": 300,
                      "keepalive_interval": 60,
                      "keepalive_probes": 5
                    }
                  }
                }
              ]
            }
          }
          EOF
    }
  }
}

However, the above envoy config is not getting reflected when I hit /clusters endpoint of envoy.