hello,
I’m working on Consul 1.20 on a Kubernetes cluster.
I have a problem that I can’t solve: all requests are blocked at 5 minutes by a timeout, and I receive a 504 error.
My goal is to have requests every hour (I know, it might seem strange, but that’s what I need).
I have a global proxy-defaults.yaml
ApiVersion: consul.hashicorp.com/v1alpha1
kind: ProxyDefaults
metadata:
name: global
namespace: consul
spec:
config:
local_request_timeout_ms: 3600000
local_idle_timeout_ms: 3600000
and for each service:
a service-defaults.yaml and
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
name: my-service
namespace: consul
spec:
protocol: http
localRequestTimeoutMs : 3600000
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceResolver
metadata:
name: my-service
namespace: consul
spec:
requestTimeout: 3600s
Regarding Envoy documentation, I identify the only one default timeout that’s set to 5 minutes is : stream_idle_timeout ( How do I configure timeouts? — envoy 1.37.0-dev-346db8 documentation )
I was looking for how to override this timeout, and in my research I came across this post that talks about the same problem: Override stream_idle_timeout 5 min default on envoy , someone pushed this PR : Adds idle_timeout envoy proxy local app config knob by johnalotoski · Pull Request #14418 · hashicorp/consul · GitHub on the Consul repository and it was released in v1.14 (as a reminder, I’m on Consul 1.20)
In the code, we can see :
// LocalIdleTimeoutMs is the number of milliseconds a request's stream to the
// local app instance may be idle. If not set, no value is set, Envoy defaults
// are respected, and an Envoy stream_idle_timeout (5m) will apply. If set,
// this LocalIdleTimeoutMs value will override the Envoy stream_idle_timeout.
main ← oulman:oulman-add-idle-timeout
opened 05:11AM - 25 Aug 22 UTC
### Description
This PR adds support for configuring the Envoy [route level i… dle_timeout](https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/route/v3/route_components.proto#envoy-v3-api-field-config-route-v3-routeaction-idle-timeout) for service-routers and local_app. For our teams scenario, this will let us extend the timeout for long-running gRPC services so that Envoy doesn't terminate the connection prematurely.
For service-routers a `IdleTimeout` parameter has been added, and for local_app a proxy config `local_idle_timeout_ms` parameter was created. I followed the [RequestTimeout](https://www.consul.io/docs/connect/config-entries/service-router#requesttimeout) and [local_request_timeout_ms](https://www.consul.io/docs/connect/proxies/envoy#local_request_timeout_ms) definitions closely.
### Testing & Reproduction steps
#### Manual Testing Steps
##### service-router IdleTimeout
Created a service-router and configured the `IdleTimeout` to 60s
```
Kind = "service-router"
Name = "waitserver"
Routes = [
{
Destination {
Service = ""
RequestTimeout = "60m"
IdleTimeout = "60s"
}
}
]
```
Dumped the Envoy config and verified that the `idle_timeout` configuration parameter was set
```
<snip>
"name": "waitserver",
"domains": [
"*"
],
"routes": [
{
"match": {
"prefix": "/"
},
"route": {
"cluster": "waitserver.default.dc1.internal.ba4be18f-a3c7-7e2f-eff4-a25a8a1fce57.consul",
"timeout": "3600s",
"idle_timeout": "60s"
}
},
<snip>
```
##### local_idle_timeout_ms
Configure `local_idle_timeout_ms` on a service proxy.config
```
service {
name = "waitserver"
id = "waitserver"
address = "10.5.0.20"
port = 9222
connect {
sidecar_service {
port = 10201
proxy {
config {
local_request_timeout_ms = 0
local_idle_timeout_ms = 0
}
}
check {
name = "Connect Envoy Sidecar"
tcp = "10.5.0.20:10201"
interval ="10s"
}
}
}
}
```
Dumped the Envoy config and verified that the `idle_timeout` configuration parameter was set on the public_listener route for `local_app`
```
{
"route_config": {
"@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"name": "public_listener",
"virtual_hosts": [
{
"name": "public_listener",
"domains": [
"*"
],
"routes": [
{
"match": {
"prefix": "/"
},
"route": {
"cluster": "local_app",
"timeout": "0s",
"idle_timeout": "0s"
}
}
]
}
]
},
```
### Links
Include any links here that might be helpful for people reviewing your PR (Tickets, GH issues, API docs, external benchmarks, tools docs, etc). If there are none, feel free to delete this section.
Please be mindful not to leak any customer or confidential information. HashiCorp employees may want to use our internal URL shortener to obfuscate links.
### PR Checklist
* [ ] updated test coverage
* [ ] external facing docs updated
* [ ] not a security concern
As written above, I put this value in the default proxy.
As its explained in PR Conversation ( Add support for configuring Envoys route idle_timeout by oulman · Pull Request #14340 · hashicorp/consul · GitHub ),
I tried as well, to add a service router on my-service :
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceRouter
metadata:
name: my-service
spec:
routes:
- destination:
idleTimeout: 3600s
requestTimeout: 3600s
service: ''
but after dumping config of the envoy of my-service i didn’t find any occurence of idle timeout.
I’m still blocked and my objective is to have requests between services that can reach one hour.
What’s wrong, what should I do?
Please could you help me to solve this issue.
Thank you in advance.
Regards