# Gateway Timeout Connection problem with Consul in non-production **URL:** \ **Category:** Consul\ **Created:** [September 5, 2023, 7:03am UTC](https://discuss.hashicorp.com/t/gateway-timeout-connection-problem-with-consul-in-non-production/57948 "2023-09-05T07:03:21Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![devika.pujari](https://avatars.discourse-cdn.com/v4/letter/d/94ad74/32.png) [@devika.pujari](https://discuss.hashicorp.com/u/devika.pujari)\ **Post date:** [September 5, 2023, 7:03am UTC](https://discuss.hashicorp.com/t/gateway-timeout-connection-problem-with-consul-in-non-production/57948/1 "2023-09-05T07:03:21Z")
Hi , We have deployed consul 1.14.1 on our k8s on-prem cluster. we have been getting below error logs for few applications using consul. Below are the error message, **at Consul.GetRequest`1.Execute(CancellationToken ct)** \*\* at Winton.Extensions.Configuration.Consul.ConsulConfigurationProvider.GetKvPairs(Boolean waitForChange, CancellationToken cancellationToken)\*\* \*\* at Winton.Extensions.Configuration.Consul.ConsulConfigurationProvider.PollingLoop(CancellationToken cancellationToken)\*\* **Consul.ConsulRequestException: Unexpected response, status code GatewayTimeout:** # **504 Gateway Time-out** **The server didn’t respond in time.** \*\* at Consul.GetRequest`1.Execute(CancellationToken ct)\*\* \*\* at Winton.Extensions.Configuration.Consul.ConsulConfigurationProvider.GetKvPairs(Boolean waitForChange, CancellationToken cancellationToken)\*\* \*\* at Winton.Extensions.Configuration.Consul.ConsulConfigurationProvider.PollingLoop(CancellationToken cancellationToken)\*\* **Consul.ConsulRequestException: Unexpected response, status code TooManyRequests: Your IP is issuing too many concurrent connections, please rate limit your calls** Please guide . ---
**Author:** ![Ranjandas](https://avatars.discourse-cdn.com/v4/letter/r/90ced4/32.png) [@Ranjandas](https://discuss.hashicorp.com/u/Ranjandas)\ **Post date:** [September 5, 2023, 10:29am UTC](https://discuss.hashicorp.com/t/gateway-timeout-connection-problem-with-consul-in-non-production/57948/2 "2023-09-05T10:29:34Z")
Hi @devika.pujari, Welcome to the HashiCorp Forums! The error you shared happens when your IP address issues too many requests. You can refer to this topic in which I have explained more about this feature in Consul. ref: [Http\_max\_conns\_per\_client in Consul Clients VS Consul servers](https://discuss.hashicorp.com/t/http-max-conns-per-client-in-consul-clients-vs-consul-servers/57560) The relevant configuration option to tweak this is the `limits.http_max_conns_per_client`. ref: [Agents - Configuration File Reference | Consul | HashiCorp Developer](https://developer.hashicorp.com/consul/docs/agent/config/config-files#http_max_conns_per_client) On Consul-K8S cluster, you will have to configure this using the `server.extraConfig` option, as this configuration is not directly exposed in the Helm chart to be configured. ref: [Helm Chart Reference | Consul | HashiCorp Developer](https://developer.hashicorp.com/consul/docs/k8s/helm#v-server-extraconfig) I hope this helps. ---
**Author:** ![monica.k](https://avatars.discourse-cdn.com/v4/letter/m/73ab20/32.png) [@monica.k](https://discuss.hashicorp.com/u/monica.k)\ **Post date:** [September 6, 2023, 6:52am UTC](https://discuss.hashicorp.com/t/gateway-timeout-connection-problem-with-consul-in-non-production/57948/3 "2023-09-06T06:52:31Z")
Hi @Ranjandas Tried the above suggestion, the helm upgrade was success but the pods are crashing: helm upgrade consul hashicorp/consul --set server.extraConfig.limits.http\_max\_conns\_per\_client=400 --kubeconfig retail-user-cluster-kubeconfig-26072023-consul --namespace consul --kube-insecure-skip-tls-verify coalesce.go:223: warning: destination for consul.server.extraConfig is a table. Ignoring non-table value ({} ) Error: UPGRADE FAILED: template: consul/templates/server-statefulset.yaml:115:51: executing “consul/templates/server-statefulset.yaml” at \: error calling include: template: consul/templates/server-config-configmap.yaml:96:14: executing "consul/templates/server-config-configmap.yaml" at \<.Values.server.extraConfig\>: wrong type for value; expected string; got map[string]interface {} ubuntu@ievxantnpw01:~/vmie-anthos-non-prod/configs helm upgrade consul hashicorp/consul --set-string server.extraConfig=“limits:\n http\_max\_conns\_per\_client: 400” --kubeconfig retail-user-cluster-kubeconfig-26072023-consul --namespace consul --kube-insecure-skip-tls-verify Release “consul” has been upgraded. Happy Helming! NAME: consul LAST DEPLOYED: Wed Sep 6 06:39:26 2023 NAMESPACE: consul STATUS: deployed REVISION: 2 NOTES: Thank you for installing HashiCorp Consul! Your release is named consul. Below are the pod status: - r get pods -n consul NAME READY STATUS RESTARTS AGE consul-consul-connect-injector-94fc6ccbc-sr8jr 0/1 CrashLoopBackOff 6 (2m33s ago) 11m consul-consul-server-0 0/1 CrashLoopBackOff 6 (39s ago) 6m50s consul-consul-webhook-cert-manager-794c9fb954-rsbgz 1/1 Running 0 11m - r logs consul-consul-server-0 -n consul Defaulted container “consul” out of: consul, locality-init (init) ==\> failed to parse /consul/config/…2023\_09\_06\_06\_43\_52.3493422559/extra-from-values.json: invalid character ‘l’ looking for beginning of value Please guide. Regards, Monica ---
**Author:** ![monica.k](https://avatars.discourse-cdn.com/v4/letter/m/73ab20/32.png) [@monica.k](https://discuss.hashicorp.com/u/monica.k)\ **Post date:** [September 6, 2023, 8:05am UTC](https://discuss.hashicorp.com/t/gateway-timeout-connection-problem-with-consul-in-non-production/57948/4 "2023-09-06T08:05:01Z")
Hi @Ranjandas, Successfully added the extra config, and upgraded the helm deployment, post upgrade while trying to access - getting default backend 404 Please suggest. Regards, Monica ---
**Author:** ![monica.k](https://avatars.discourse-cdn.com/v4/letter/m/73ab20/32.png) [@monica.k](https://discuss.hashicorp.com/u/monica.k)\ **Post date:** [September 14, 2023, 8:32am UTC](https://discuss.hashicorp.com/t/gateway-timeout-connection-problem-with-consul-in-non-production/57948/5 "2023-09-14T08:32:37Z")
Hi, The workaround hasn’t helped, could someone help here please? Regards, Monica ---
**Author:** ![Ranjandas](https://avatars.discourse-cdn.com/v4/letter/r/90ced4/32.png) [@Ranjandas](https://discuss.hashicorp.com/u/Ranjandas)\ **Post date:** [September 16, 2023, 10:19am UTC](https://discuss.hashicorp.com/t/gateway-timeout-connection-problem-with-consul-in-non-production/57948/6 "2023-09-16T10:19:43Z")
Hi @monica.k, The configuration change recommended was to prevent HTTP 429s from happening, due to HTTP ratelimiting. The HTTP 404 that you are getting could be due to trying to access a resource that doesn’t exist in Consul. For eg: ```auto curl -H "X-CONSUL-TOKEN: root" -s 0:8500/v1/kv/this-kv-does-not-exist -i HTTP/1.1 404 Not Found Vary: Accept-Encoding X-Consul-Default-Acl-Policy: deny X-Consul-Index: 1 X-Consul-Knownleader: true X-Consul-Lastcontact: 0 X-Consul-Query-Backend: blocking-query Date: Sat, 16 Sep 2023 10:12:54 GMT Content-Length: 0 ``` Sharing the `DEBUG` level logs from the agent where your application is connecting will help understand which request is resulting in HTTP 404s. ---
**Author:** ![Ranjandas](https://avatars.discourse-cdn.com/v4/letter/r/90ced4/32.png) [@Ranjandas](https://discuss.hashicorp.com/u/Ranjandas)\ **Post date:** [September 19, 2023, 7:28am UTC](https://discuss.hashicorp.com/t/gateway-timeout-connection-problem-with-consul-in-non-production/57948/8 "2023-09-19T07:28:19Z")
Hi @monica.k Are you accessing Consul via a LoadBalancer? From your previous message, I can see that your `http_max_conns_per_client` is set to `400`. If you are still seeing the error, that means the number of connections are exceeding the configured value. You can follow this KB article to tune the configuration. Towards the end of the KB, you will see the use of `lsof` command to count the number of active connections. Use that to find the right max settings for the number of HTTP connections per client. ref: [Consul http\_max\_conns\_per\_client tuning – HashiCorp Help Center](https://support.hashicorp.com/hc/en-us/articles/7418803362963-Consul-http-max-conns-per-client-tuning) I hope this helps. ---
**Author:** ![monica.k](https://avatars.discourse-cdn.com/v4/letter/m/73ab20/32.png) [@monica.k](https://discuss.hashicorp.com/u/monica.k)\ **Post date:** [September 19, 2023, 12:53pm UTC](https://discuss.hashicorp.com/t/gateway-timeout-connection-problem-with-consul-in-non-production/57948/9 "2023-09-19T12:53:25Z")
Hi @Ranjandas, the above issue is resolved, in our applications that are using consul, we are seeing the below logs: at Consul.GetRequest`1.Execute(CancellationToken ct) at Winton.Extensions.Configuration.Consul.ConsulConfigurationProvider.GetKvPairs(Boolean waitForChange, CancellationToken cancellationToken) at Winton.Extensions.Configuration.Consul.ConsulConfigurationProvider.PollingLoop(CancellationToken cancellationToken)Consul.ConsulRequestException: Unexpected response, status code GatewayTimeout: # 504 Gateway Time-out The server didn’t respond in time. Could you please suggest. Regards, Monica