Hi,
We’re seeing intermittent 503 errors from one of our Consul Connect sidecars when we run our integration tests, usually when two tests are creating large files and uploading them at roughly the same time.
The error we see in our tests
Error: unexpected response: Response { url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(127.0.0.1)), port: Some(1006), path: "/api/plugin/create", query: None, fragment: None }, status: 503, headers: {"content-length": "95", "content-type": "text/plain", "date": "Tue, 18 Oct 2022 19:32:05 GMT", "server": "envoy", "connection": "close"} }
From the Consul Connect sidecar logs:
[2022-10-18 19:32:05.508][15][debug][http] [source/common/http/conn_manager_impl.cc:1494] [C15][S4021500588519904633] Read-enabling downstream stream due to filter callbacks.
[2022-10-18 19:32:05.508][15][debug][client] [source/common/http/codec_client.cc:106] [C16] disconnect. resetting 1 pending requests
[2022-10-18 19:32:05.508][15][debug][client] [source/common/http/codec_client.cc:142] [C16] request reset
[2022-10-18 19:32:05.508][15][debug][router] [source/common/router/router.cc:1130] [C15][S4021500588519904633] upstream reset: reset reason: connection termination, transport failure reason:
[2022-10-18 19:32:05.509][15][debug][http] [source/common/http/filter_manager.cc:953] [C15][S4021500588519904633] Sending local reply with details upstream_reset_before_response_started{connection termination}
[2022-10-18 19:32:05.509][15][debug][http] [source/common/http/conn_manager_impl.cc:1467] [C15][S4021500588519904633] encoding headers via codec (end_stream=false):
':status', '503'
'content-length', '95'
'content-type', 'text/plain'
'date', 'Tue, 18 Oct 2022 19:32:05 GMT'
'server', 'envoy'
'connection', 'close'
The error appears to be upstream_reset_before_response_started{connection termination} from the Envoy proxy (It’s one of the documented Envoy error response codes). This is relatively common in Istio and usually requires tweaking keep-alive settings and/or timeouts. However, this appears to be rare with Consul so I’m uncertain what the root cause or the right way to fix this is.
- Does anyone know what the cause of this error is or have suggestions on how to further debug the error?
- What is the recommended way to update timeout and/or keep alive settings for Consul Connect sidecars as suggested in this Istio bug thread?
- Is there a way to set envoy retry policies such that it can automatically retry on reset for Consul Connect sidecars as suggested in this Istio bug thread?
Any help would be greatly appreciated