I’ve a few services that can take about 10000 requests per second. I registered ingress services and started them on two different nodes. The behavior of ingress is inconsistent. Sometimes, i’m able to push 5000 rps max for 2 minutes. The next run gets me a gateway timeout error. I was expecting to reach 7000 rps or so if I run load tests from the two ingress nodes but, no, it is lower than one node. If I take ingress out of the picture and hit the service directly, i can get to about 10000 rps w/o any issues. I see overflow errors in envoy admin logs. I’ll post the debug entries from envoy admin logs shortly. However, i’m not able to increase max connections or concurrent sessions for ingress envoy since escape hatch does not work and default values are at 1024. Did anyone else run into this kind of a problem where ingress gateway is not performing as it should?
Here are the errors I see after I enable envoy debug:
[2020-11-30 22:14:00.016][1786462][debug][pool] [external/envoy/source/common/conn_pool/conn_pool_base.cc:190] max pending streams overflow
[2020-11-30 22:14:00.016][1786462][debug][router] [external/envoy/source/common/router/router.cc:1031] [C4979][S16595666839539267465] upstream reset: reset reason: overflow, transport failure reason:
[2020-11-30 22:14:00.016][1786462][debug][http] [external/envoy/source/common/http/filter_manager.cc:805] [C4979][S16595666839539267465] Sending local reply with details upstream_reset_before_response_started{overflow}
[2020-11-30 22:14:00.016][1786462][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:1435] [C4979][S16595666839539267465] encoding headers via codec (end_stream=false):
‘:status’, ‘503’
‘x-envoy-overloaded’, ‘true’
‘content-length’, ‘81’
‘content-type’, ‘text/plain’
‘date’, ‘Mon, 30 Nov 2020 22:13:59 GMT’
‘server’, ‘envoy’
[2020-11-30 22:14:00.016][1786462][debug][router] [external/envoy/source/common/router/router.cc:1178] [C1619][S5188761523082112748] upstream headers complete: end_stream=false
[2020-11-30 22:14:00.016][1786462][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:1435] [C1619][S5188761523082112748] encoding headers via codec (end_stream=false):