ZeDKoN
October 1, 2021, 9:58am
1
Hi there,
I would like to know if someone has successfully overridden the property stream_idle_timeout
on envoy side car proxy? I was trying to do it without having to rewrite the listeners, routes generated by consul.
I have unsuccessfully tried to use the escape hatch overrides
Any example, or help would be really appreciated.
Thanks
I ran into needing to do this as well for some long running APIs.
I got Nomad quite close to working with an escape hatch override, but the blocker seems to be that Nomad interpolation isn’t available in the escape hatch snippet to pass in the dynamic port. If you aren’t using Nomad and have just a static port assignment for the public service listener, you should be able to use the snippet with your port substituted in this issue to make it work:
opened 04:48PM - 30 Aug 22 UTC
type/bug
### Nomad version
Nomad v1.2.9
### Issue
Envoy uses a default [stream_idle_… timeout](https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto#envoy-v3-api-field-extensions-filters-network-http-connection-manager-v3-httpconnectionmanager-stream-idle-timeout) of 5 minutes which is in conflict with a few long running APIs we are using. There are a few direct [override knobs](https://www.consul.io/docs/connect/proxies/envoy#local_request_timeout_ms) for envoy timeouts in Consul, but there is no direct knob for this `stream_idle_timeout`. Advanced Consul Escape hatches are [available](https://www.consul.io/docs/connect/proxies/envoy#advanced-configuration), and in this case, configuring an escape hatch override for [envoy_public_listener_json](https://www.consul.io/docs/connect/proxies/envoy#envoy_public_listener_json) appears that it would solve the problem, as we could add a route public listener `idle_timeout` which will override envoy's default `stream_idle_timeout`, or directly modify the `stream_idle_timeout` itself.
However, when Nomad sets up the consul job, the dynamic port the envoy listener will use ahead of time is not known, so some Nomad interpolation appears to be necessary to be able to declare this snippet properly. Taking the example of the Consul `envoy_public_listener_json` from the Consul link above, with slight modifications, I believe we'd need to be able to set the `connect.sidecar_service.proxy.config.envoy_public_listener_json` stanza in the Nomad job declaration to something like the following where the Nomad assigned port is interpolated and passed to the Consul escape hatch json override:
```
{
"@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "public_listener:0.0.0.0:${NOMAD_PORT_connect_proxy_<SERVICE_NAME>}",
"address": {
"socket_address": {
"address": "0.0.0.0",
"port_value": ${NOMAD_PORT_connect_proxy_<SERVICE_NAME>}
}
},
"filter_chains": [
{
"filters": [
{
"name": "envoy.filters.network.http_connection_manager",
"typed_config": {
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
"stat_prefix": "public_listener",
"http_filters": [
{
"name": "envoy.filters.http.router"
}
],
"route_config": {
"name": "public_listener",
"virtual_hosts": [
{
"name": "public_listener",
"domains": ["*"],
"routes": [
{
"match": {
"prefix": "/"
},
"route": {
"cluster": "local-app",
"idle_timeout": "10m"
}
}
]
}
]
}
}
}
]
}
],
"traffic_direction": "INBOUND"
}
```
However, interpolation of the Nomad assigned dynamic port to the Consul connect service doesn't appear available to the passed json escape hatch override in the testing I've done, which seems to eliminate the possibility of using envoy escape hatches to override parameters there aren't already direct Consul overrides for.
I've tested the above approach by deploying the job with a random hardcoded port substituted in the parameterization above for the escape hatch which will be incorrect. Then, once the job is deployed, adjusting the above job definition for the escape hatch snippet and assigning the correct port that Nomad has utilized for the connect proxy and re-deploying/updating the job, at which point the escape hatch override does work as intended. So it appears the only remaining issue is the ability to interpolate a Nomad dynamic port and pass it to the escape hatch snippet.
Perhaps Nomad interpolation can done in this snippet and I'm not aware of it?
Since that’s not an option for us at the moment, I patched in a config knob for an envoy idle_timeout which will override the default stream_idle_timeout when set:
hashicorp:main
← johnalotoski:local-idle-timeout
opened 02:26PM - 31 Aug 22 UTC
### Description
We require override of envoy's default `stream_idle_timeout` wh… ich we are unable to do via Nomad and escape hatches at this time AFAICT. This has also been asked for in the community. See refs below.
### Links
* https://github.com/hashicorp/nomad/issues/14403
* https://discuss.hashicorp.com/t/override-stream-idle-timeout-5-min-default-on-envoy/30175
### PR Checklist
* [ ] updated test coverage
* [X] external facing docs updated
* [X] not a security concern
Also, we are using this same patch in 1.11.2 as well, with the diff file being slightly different and seen in this commit here:
committed 07:19PM - 30 Aug 22 UTC