Upstream connection timeout does not work

Consul
,
1

We have service upstream configured and connection timeout defined as per documents here:

service {
  name = "cpanel"
  port = 4000
  connect {
    sidecar_service {
      proxy {
        upstreams = [
          {
            destination_name = "gateway-internal-api"
            local_bind_port  = 10000
            config {
              connect_timeout_ms = 10000
              }
          },
        ]
      }
    }
  }

But we have not managed to adjust default connection timeout, which is obviously set to 5s. Even we set timeout to 10s, it still timeouts after 5s.
How we can adjust default config value for timeout for upstream?
We are running following Consul and Envoy versions:

Consul v1.8.3
Envoy 1.14.2
2

@tmiroslav I fired this question internally but I think this was addressed in Consul 1.10.0. Here is the GitHub issue I am thinking off. I’ll let you know what I hear.

3

See Envoy/Consul Connect - upstream request timeout - Nomad - HashiCorp Discuss as well.

4

@karl-cardenas-coding In my use case, Consul/Envoy is running on VMs. So, not using Nomad or containerized env. On github issue you posted, it’s close to what I need. But, it’s about Envoy timeout which is 15s by default. In my case timeout is 5s. My app, which is using Consul to connect, timeouts exactly after 5s. This is what in the logs when timeouts happened.

[error] GET http://localhost:10000/api/v1/pools/95364/user_sessions/ -> error: :timeout (5001.191 ms)

And connect_timeout_ms proxy upstream config parameters from the link I posed above is exactly why it timeouts I think. But, as I sad, when I try this way to increase this timeout value, i did not work, at least not in Consul/Envoy versions I am using. Can you please check if this would work in Consul 1.10.0? So we might plan to go with Consul/Envoy upgrade. Or, if need to do anything else to increase this timeout value?
Thank you!

5

Just a longshot:
Shouldn’t the config-block be inline with the upstreams-block? E.g. like this example

proxy {
  upstreams = [{
    destination_name = "gateway-internal-api"
    local_bind_port  = 10000
  }]
  config {
    connect_timeout_ms = 10000
  }
}

EDIT: Ok, I was wrong, I guess… Not at all confusing… :wink:

6

Hi @karl-cardenas-coding
Have you managed to check with the team if I should upgrade to Consul 1.10.0, so this timeout increase to work?
I was suggested to set local_connect_timeout_ms in proxy config definition of service that my service, which timeouts, talked to. Like following:

service {
	name = "gateway-internal-api"
	port = 8787
	connect {
	  sidecar_service {
		proxy {
		  config {
			 local_connect_timeout_ms = 10000
		  }
		  upstreams = [
			....
		  ]
		}
	  }
	}

So, my service which timeouts talks to gateway-internal-api service.
But still it timeouts after 5s. Do I need to upgrade, or setup config differently.
Thanx!
Miroslav

7

Hi @tmiroslav I fired off another message to engineering, but with the holidays it’s a bit challenging to get a hold of someone. Is it possible for you to test with a newer version, or is that a bit challenging due to environment restrictions ?

8

Hi @karl-cardenas-coding

It’s bit challenging for me to setup test environment as I do have env conditions for timeouts in production only. In dev env. it’s very hard to reproduce an issue. We can schedule and plan upgrade, but should now if this is going to fix timeout issue. We would not upgrade otherwise.
Please let me know if any feedback from engineering team about this.

Thank you!

BR,
Miroslav