Envoy sidecar terminated upstream connection after 5 minutes of inactivity

Consul
1

Hi All,

I hope I can get anwser on this issue.

My current Consul mesh version: 1.20.0

My test setup:

  1. Service A (downstream) is calling service B (upstream)

  2. requestTimeout for ServiceRouter of the service B is set at high value (10 minutes for this test)

  3. localRequestTimeoutMs for ServiceDefaults of the service B is set at high value (10 minutes for this test)

  4. To simulate long running request, I put a sleep of more than 5 minutes for service B

Test result:

  1. Envoy always terminated the upstream connection at exact 5 minutes.

  2. I am aware of that Envoy’s default stream idle timeout is 5 minutes. (stream_idle_timeout attribute)

To mitigate the issue:

  1. I updated ServiceRouter of Service B to add : idleTimeout: 600s

  2. I did a dump of service A (downstream) envoy and confirmed that the idle_timeout of 600s is reflected under the route_config → route for the Service B (upstream)

  3. However, the Envoy still terminated the upstream connection at 5 minutes.

My question is:

  1. Adding idleTimeout under ServiceRouter of service B is the correct action to override the default stream_idle_timeout of the Envoy?

  2. If that is the case then why it is not taking any effect?

Many thanks!

~K

2

Some updates:

  • More tests with idleTimeout set with values less than 5 minutes for service router confirmed working.
  • This leads me to believe that 5 minutes constraint in this case is coming from the stream_idle_timeout set between the Envoy’s sidecar of the ServiceB and the Service B local application.
  • Unfortunately, I have no way of adding the idleTimeout under the ServiceDefaults of the Service B. Seems like Consul does not support it.