Hi ,
We have deployed consul 1.14.1 on our k8s on-prem cluster. we have been getting below error logs for few applications using consul.
Below are the error message,
at Consul.GetRequest`1.Execute(CancellationToken ct)
** at Winton.Extensions.Configuration.Consul.ConsulConfigurationProvider.GetKvPairs(Boolean waitForChange, CancellationToken cancellationToken)**
** at Winton.Extensions.Configuration.Consul.ConsulConfigurationProvider.PollingLoop(CancellationToken cancellationToken)**
Consul.ConsulRequestException: Unexpected response, status code GatewayTimeout:
504 Gateway Time-out
The server didn’t respond in time.
** at Consul.GetRequest`1.Execute(CancellationToken ct)**
** at Winton.Extensions.Configuration.Consul.ConsulConfigurationProvider.GetKvPairs(Boolean waitForChange, CancellationToken cancellationToken)**
** at Winton.Extensions.Configuration.Consul.ConsulConfigurationProvider.PollingLoop(CancellationToken cancellationToken)**
Consul.ConsulRequestException: Unexpected response, status code TooManyRequests: Your IP is issuing too many concurrent connections, please rate limit your calls
Please guide .
Hi @devika.pujari,
Welcome to the HashiCorp Forums!
The error you shared happens when your IP address issues too many requests. You can refer to this topic in which I have explained more about this feature in Consul.
ref: Http_max_conns_per_client in Consul Clients VS Consul servers
The relevant configuration option to tweak this is the limits.http_max_conns_per_client.
ref: Agents - Configuration File Reference | Consul | HashiCorp Developer
On Consul-K8S cluster, you will have to configure this using the server.extraConfig option, as this configuration is not directly exposed in the Helm chart to be configured.
ref: Helm Chart Reference | Consul | HashiCorp Developer
I hope this helps.
Hi @Ranjandas
Tried the above suggestion, the helm upgrade was success but the pods are crashing:
helm upgrade consul hashicorp/consul --set server.extraConfig.limits.http_max_conns_per_client=400 --kubeconfig retail-user-cluster-kubeconfig-26072023-consul --namespace consul --kube-insecure-skip-tls-verify
coalesce.go:223: warning: destination for consul.server.extraConfig is a table. Ignoring non-table value ({}
)
Error: UPGRADE FAILED: template: consul/templates/server-statefulset.yaml:115:51: executing “consul/templates/server-statefulset.yaml” at <include (print .Template.BasePath "/server-config-configmap.yaml") .>: error calling include: template: consul/templates/server-config-configmap.yaml:96:14: executing "consul/templates/server-config-configmap.yaml" at <.Values.server.extraConfig>: wrong type for value; expected string; got map[string]interface {}
ubuntu@ievxantnpw01:~/vmie-anthos-non-prod/configs helm upgrade consul hashicorp/consul --set-string server.extraConfig=“limits:\n http_max_conns_per_client: 400” --kubeconfig retail-user-cluster-kubeconfig-26072023-consul --namespace consul --kube-insecure-skip-tls-verify
Release “consul” has been upgraded. Happy Helming!
NAME: consul
LAST DEPLOYED: Wed Sep 6 06:39:26 2023
NAMESPACE: consul
STATUS: deployed
REVISION: 2
NOTES:
Thank you for installing HashiCorp Consul!
Your release is named consul.
Below are the pod status:
-
r get pods -n consul
NAME READY STATUS RESTARTS AGE
consul-consul-connect-injector-94fc6ccbc-sr8jr 0/1 CrashLoopBackOff 6 (2m33s ago) 11m
consul-consul-server-0 0/1 CrashLoopBackOff 6 (39s ago) 6m50s
consul-consul-webhook-cert-manager-794c9fb954-rsbgz 1/1 Running 0 11m
-
r logs consul-consul-server-0 -n consul
Defaulted container “consul” out of: consul, locality-init (init)
==> failed to parse /consul/config/…2023_09_06_06_43_52.3493422559/extra-from-values.json: invalid character ‘l’ looking for beginning of value
Please guide.
Regards,
Monica
Hi @Ranjandas,
Successfully added the extra config, and upgraded the helm deployment, post upgrade while trying to access - getting default backend 404
Please suggest.
Regards,
Monica
Hi,
The workaround hasn’t helped, could someone help here please?
Regards,
Monica
Hi @monica.k,
The configuration change recommended was to prevent HTTP 429s from happening, due to HTTP ratelimiting. The HTTP 404 that you are getting could be due to trying to access a resource that doesn’t exist in Consul.
For eg:
curl -H "X-CONSUL-TOKEN: root" -s 0:8500/v1/kv/this-kv-does-not-exist -i
HTTP/1.1 404 Not Found
Vary: Accept-Encoding
X-Consul-Default-Acl-Policy: deny
X-Consul-Index: 1
X-Consul-Knownleader: true
X-Consul-Lastcontact: 0
X-Consul-Query-Backend: blocking-query
Date: Sat, 16 Sep 2023 10:12:54 GMT
Content-Length: 0
Sharing the DEBUG level logs from the agent where your application is connecting will help understand which request is resulting in HTTP 404s.
Hi @Ranjandas,
Applied the workaround, the consul is up and running yet the same error message:
Hi @monica.k
Are you accessing Consul via a LoadBalancer? From your previous message, I can see that your http_max_conns_per_client is set to 400.
If you are still seeing the error, that means the number of connections are exceeding the configured value.
You can follow this KB article to tune the configuration. Towards the end of the KB, you will see the use of lsof command to count the number of active connections. Use that to find the right max settings for the number of HTTP connections per client.
ref: Consul http_max_conns_per_client tuning – HashiCorp Help Center
I hope this helps.
Hi @Ranjandas, the above issue is resolved, in our applications that are using consul, we are seeing the below logs:
at Consul.GetRequest`1.Execute(CancellationToken ct) at Winton.Extensions.Configuration.Consul.ConsulConfigurationProvider.GetKvPairs(Boolean waitForChange, CancellationToken cancellationToken) at Winton.Extensions.Configuration.Consul.ConsulConfigurationProvider.PollingLoop(CancellationToken cancellationToken)Consul.ConsulRequestException: Unexpected response, status code GatewayTimeout:
504 Gateway Time-out
The server didn’t respond in time.
Could you please suggest.
Regards,
Monica
