# Consul agent CPU Utilization jumping to its limits in 5 mins with bare minimum traffic and not getting recovered **URL:** \ **Category:** Consul\ **Tags:** k8s\ **Created:** [May 3, 2023, 11:43am UTC](https://discuss.hashicorp.com/t/consul-agent-cpu-utilization-jumping-to-its-limits-in-5-mins-with-bare-minimum-traffic-and-not-getting-recovered/53377 "2023-05-03T11:43:07Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![shashankgarg22](https://sea2.discourse-cdn.com/hashicorp/user_avatar/discuss.hashicorp.com/shashankgarg22/32/35498_2.png) [@shashankgarg22](https://discuss.hashicorp.com/u/shashankgarg22)\ **Post date:** [May 3, 2023, 11:43am UTC](https://discuss.hashicorp.com/t/consul-agent-cpu-utilization-jumping-to-its-limits-in-5-mins-with-bare-minimum-traffic-and-not-getting-recovered/53377/1 "2023-05-03T11:43:07Z")
### Question: Consul agent CPU Utilization jumping to its limits in 5 mins with bare minimum traffic and not getting recovered ### Problem Description: We are using Consul for service discovery. Our consul servers with 3 Replica set, each pod is having consul agent as a separate container. Service discovery is done through those consul agent containers. Sporadically, we are observing consul agent containers reaching their CPU Limits which are defined as 200m in 5 mins with bare minimum traffic as : **Monitor Logs** 2023-04-27T13:44:47.264Z [DEBUG] agent.client.memberlist.lan: memberlist: Stream connection from=xxx.xxx.xxx.xxx:59022 2023-04-27T13:44:36.329Z [DEBUG] agent: Check status updated: check=service:mtb123-LocalD-mtb123-scscf-8488d6b876-xkdjl status=passing 2023-04-27T13:44:39.099Z [DEBUG] agent: Check status updated: check=service:mtb123-RtpRestServ-mtb123-scscf-8488d6b876-xkdjl status=passing 2023-04-27T13:44:39.624Z [DEBUG] agent.http: Request finished: method=GET url=/v1/status/leader from=127.0.0.1:50022 latency=365.303µs 2023-04-27T13:42:33.121Z [DEBUG] agent: Service in sync: service=mtb123-LocalD-mtb123-scscf-8488d6b876-xkdjl 2023-04-27T13:42:33.121Z [DEBUG] agent: Service in sync: service=mtb123-RtpRestServ-mtb123-scscf-8488d6b876-xkdjl 2023-04-27T13:42:33.121Z [DEBUG] agent: Check in sync: check=service:mtb123-RtpRestServ-mtb123-scscf-8488d6b876-xkdjl Most of the time the CPU remains less than 30m - 40m. **Once it reaches 200m then it is not coming back to its original state.** **After having the top command** wnv4a0cscf0001c-scscf-66785f456f-vqgwj consul-agent **201m** 69Mi **Consul version** : 1.8.0 ### **Consul Helm chart template:** consul-agent: ```auto **Limits:** **cpu: 200m** memory: 200Mi Requests: cpu: 100m memory: 200Mi **Readiness:** exec [/bin/sh -ec curl --connect-timeout 5 --max-time 5 http://localhost:8500/v1/status/leader \ ``` 2\>/dev/null grep -E ‘“.+”’ ] delay=0s timeout=5s period=10s #success=1 #failure=3 ### **Events:** Type Reason Age From Message * * * **Warning Unhealthy 23s (x19473 over 34d) kubelet, wnv4a-c00-perfworkerbm-22 Readiness probe failed:** ### **LOGS FOR THE CONSUL\_AGEN CONTAINER** [wnv4a0cscf0001c-user@wnv4b0depl0001vm001 ~] [wnv4a0cscf0001c-user@wnv4b0depl0001vm001 ~] **[wnv4a0cscf0001c-user@wnv4b0depl0001vm001 ~]$ kc logs wnv4a0cscf0001c-scscf-66785f456f-vqgwj -c consul-agent /opt/startups/functions\_source.sh: line 7: /logstore/wnv4a0cscf0001c-scscf-66785f456f-vqgwj/consul-agent/startlog.txt: No such file or directory** **Consul in Client Mode** Cannot create /tmp/consul-test. Reverting to /tmp ==\> Starting Consul agent… Version: ‘v1.0.6.xxxx-23-g6e7b310’ Node ID: ‘ffb41e1d-6a01-xxxxx-32d3-35864ca780d6’ Node name: ‘wnv4a0cscf0001c-scscf-66785f456f-vqgwj’ Datacenter: ‘wnv4a0cscf0001c’ (Segment: ‘’) Server: false (Bootstrap: false) Client Addr: [127.0.0.1 xxx.xx.xx.229] (HTTP: 8500, HTTPS: -1, gRPC: -1, DNS: -1) Cluster Addr: xx.xx.xx.229 (LAN: 8301, WAN: 8302) Encrypt: Gossip: true, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false **==\> Log data will now stream in as it occurs:** ==\> Consul agent running! 2023-03-21T18:11:41.829Z [ERROR] agent.anti\_entropy: failed to sync remote state: error=“No known Consul servers” 2023-03-21T18:11:48.429Z [ERROR] agent: Failed to check for updates: error=“Get “[https://checkpoint-api.hashicorp.com/v1/check/consul?arch=amd64&os=linux&signature=446d8fb7-xxxc&version=1.8.0](https://checkpoint-api.hashicorp.com/v1/check/consul?arch=amd64&os=linux&signature=446d8fb7-xxxc&version=1.8.0)”: dial tcp 18.160.60.92:443: i/o timeout (Client.Timeout exceeded while awaiting headers)” 2023-03-21T18:38:24.791Z [ERROR] agent.client: RPC failed to server: method=Status.Leader server=xxx.xx.xx.123:8300 error=“rpc error making call: stream closed” 2023-03-21T18:38:33.012Z [ERROR] agent.client.memberlist.lan: memberlist: Conflicting address for wnv4a0cscf0001c-sd-0. Mine: xx.xx.xxxxx.123:8301 Theirs: xx.xxx.xxx.12:8301 Old state: 2 2023-03-22T18:39:00.607Z [ERROR] agent: Failed to check for updates: error=“Get “[https://checkpoint-api.hashicorp.com/v1/check/consul?arch=amd64&os=linux&signature=4xxx8fb7-8d99-a8d6-xxxxec3d0c&version=1.8.0](https://checkpoint-api.hashicorp.com/v1/check/consul?arch=amd64&os=linux&signature=4xxx8fb7-8d99-a8d6-xxxxec3d0c&version=1.8.0)”: context deadline exceeded (Client.Timeout exceeded while awaiting headers)” [wnv4a0cscf0001c-user@wnv4b0depl0001vm001 ~] [wnv4a0cscf0001c-user@wnv4b0depl0001vm001 ~] ### Victoria matrix graph : ![image](https://global.discourse-cdn.com/hashicorp/original/3X/e/7/e762adc1d2ac27a954f2f6bf62fa9ea14e030b5e.png) ### Debug Error: ![MicrosoftTeams-image (2)](https://global.discourse-cdn.com/hashicorp/original/3X/c/a/cacb5c4fdf19bd565fa643fa90278af106bd1626.png) ### Please Help us to Solve this Problem and please describe the cause for it. \*\*If Any other details are needed, please let us know \*\*