Consul keeps high CPU utilization

Hi there,

We deployed neuvector in our k8s cluster. And the consul process is in all the neuvector pod. So I think there should be a consul cluster, right?

Then, we found the consul process consumed so much CPU.

And we found so many error logs. Following is part of them.

2020-03-18T23:11:06.533|ERRO|CTL|cache.getCommonPorts: Fail to parse - port=
2020-03-18T23:11:06.533|ERRO|CTL|cache.getCommonPorts: Fail to parse - port=
2020-03-18T23:11:06.533|ERRO|CTL|cache.getCommonPorts: Fail to parse - port=
2020-03-18T23:11:06.533|ERRO|CTL|cache.getCommonPorts: Fail to parse - port=
2020-03-18T23:11:06.533|ERRO|CTL|cache.getCommonPorts: Fail to parse - port=
    2020/03/18 23:12:48 [INFO] consul.fsm: snapshot created in 38.007µs
    2020/03/18 23:12:48 [INFO] raft: Starting snapshot up to 122048314
    2020/03/18 23:12:48 [INFO] snapshot: Creating new snapshot at /tmp/neuvector/raft/snapshots/2-122048314-1584573168213.tmp
    2020/03/18 23:12:48 [INFO] snapshot: reaping snapshot /tmp/neuvector/raft/snapshots/2-122015057-1584570214964
    2020/03/18 23:12:48 [INFO] raft: Compacting logs from 122021420 to 122038074
    2020/03/18 23:12:48 [INFO] raft: Snapshot to 122048314 complete
2020-03-18T23:22:24.872|INFO|CTL|cache.controllerUpdate: Add or update controller - controller={CLUSDevice:{ID:c60912fed4b4c222cde27c55b6105f2d814667fb6e296a70b49908c71cc8dda4 Name:k8s_neuvector-controller-pod_neuvector-controller-pod-54c4ccc89-tnd8d_default_9e9a2694-06e4-11ea-a61a-525400d4fd47_1 SelfHostname:neuvector-controller-pod-54c4ccc89-tnd8d HostName:tc-10-189-169-145 HostID:tc-10-189-169-145:5LDF:4OY6:XBPD:IIIZ:HL2Q:X73I:ZAHD:5DX2:ZB2O:C77K:AGW7:QB6O Domain:default NetworkMode:container:0fd4cc3eed9249289795d5799c8f64617f235e79585dae3bc723eb7ca484b93b PidMode: Ver:v2.5.1 Labels:map[annotation.io.kubernetes.container.hash:2be7c349 annotation.io.kubernetes.container.preStopHandler:{"exec":{"command":["/usr/local/bin/consul","leave"]}} annotation.io.kubernetes.container.restartCount:1 annotation.io.kubernetes.container.terminationMessagePath:/dev/termination-log annotation.io.kubernetes.container.terminationMessagePolicy:File annotation.io.kubernetes.pod.terminationGracePeriod:60 io.kubernetes.container.logpath:/var/log/pods/9e9a2694-06e4-11ea-a61a-525400d4fd47/neuvector-controller-pod/1.log io.kubernetes.container.name:neuvector-controller-pod io.kubernetes.docker.type:container io.kubernetes.pod.name:neuvector-controller-pod-54c4ccc89-tnd8d io.kubernetes.pod.namespace:default io.kubernetes.pod.uid:9e9a2694-06e4-11ea-a61a-525400d4fd47 io.kubernetes.sandbox.id:0fd4cc3eed9249289795d5799c8f64617f235e79585dae3bc723eb7ca484b93b neuvector.image:neuvector/controller neuvector.role:controller] CreatedAt:2020-01-02 05:49:56.751504169 +0000 UTC StartedAt:2020-01-02 05:49:56.926307661 +0000 UTC JoinedAt:2020-01-02 05:50:02.386014827 +0000 UTC MemoryLimit:0 CPUs: ClusterIP:172.16.56.41 RPCServerPort:18400 Pid:29550 Ifaces:map[eth0:[{IPNet:{IP:172.16.56.41 Mask:ffffff00} Gateway: Scope:global NetworkID: NetworkName:}]]} Leader:false OrchConnStatus:connected OrchConnLastError:}

Can someone give me some advice or tell me how to investigate it?

By the way, in the neuvector pods, the version of consul is v1.5.1.

# consul version
Consul v1.5.1
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

And the status seems good.

# consul members
Node           Address              Status  Type    Build  Protocol  DC         Segment
172.16.15.11   172.16.15.11:18301   alive   server  1.5.1  2         neuvector  <all>
172.16.33.8    172.16.33.8:18301    alive   server  1.5.1  2         neuvector  <all>
172.16.56.41   172.16.56.41:18301   alive   server  1.5.1  2         neuvector  <all>
172.16.10.13   172.16.10.13:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.11.10   172.16.11.10:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.12.11   172.16.12.11:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.13.7    172.16.13.7:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.14.8    172.16.14.8:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.15.12   172.16.15.12:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.16.8    172.16.16.8:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.17.9    172.16.17.9:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.18.14   172.16.18.14:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.19.12   172.16.19.12:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.20.12   172.16.20.12:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.21.9    172.16.21.9:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.22.8    172.16.22.8:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.23.11   172.16.23.11:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.24.9    172.16.24.9:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.25.8    172.16.25.8:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.26.10   172.16.26.10:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.27.8    172.16.27.8:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.28.10   172.16.28.10:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.29.11   172.16.29.11:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.3.11    172.16.3.11:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.30.10   172.16.30.10:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.31.10   172.16.31.10:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.32.9    172.16.32.9:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.33.9    172.16.33.9:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.34.14   172.16.34.14:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.35.9    172.16.35.9:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.36.8    172.16.36.8:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.37.17   172.16.37.17:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.38.9    172.16.38.9:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.39.18   172.16.39.18:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.4.11    172.16.4.11:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.40.9    172.16.40.9:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.41.18   172.16.41.18:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.42.11   172.16.42.11:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.43.185  172.16.43.185:18301  alive   client  1.5.1  2         neuvector  <default>
172.16.44.57   172.16.44.57:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.45.15   172.16.45.15:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.46.55   172.16.46.55:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.47.103  172.16.47.103:18301  alive   client  1.5.1  2         neuvector  <default>
172.16.48.48   172.16.48.48:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.49.63   172.16.49.63:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.5.7     172.16.5.7:18301     alive   client  1.5.1  2         neuvector  <default>
172.16.50.49   172.16.50.49:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.51.46   172.16.51.46:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.52.44   172.16.52.44:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.53.48   172.16.53.48:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.54.57   172.16.54.57:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.55.50   172.16.55.50:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.56.40   172.16.56.40:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.57.8    172.16.57.8:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.58.33   172.16.58.33:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.59.9    172.16.59.9:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.6.15    172.16.6.15:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.60.4    172.16.60.4:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.61.2    172.16.61.2:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.62.4    172.16.62.4:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.63.4    172.16.63.4:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.64.4    172.16.64.4:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.65.2    172.16.65.2:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.66.4    172.16.66.4:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.67.2    172.16.67.2:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.68.3    172.16.68.3:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.69.3    172.16.69.3:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.7.8     172.16.7.8:18301     alive   client  1.5.1  2         neuvector  <default>
172.16.70.4    172.16.70.4:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.71.4    172.16.71.4:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.72.3    172.16.72.3:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.73.2    172.16.73.2:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.74.2    172.16.74.2:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.8.10    172.16.8.10:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.9.12    172.16.9.12:18301    alive   client  1.5.1  2         neuvector  <default>

If you need more information, please let me know. Thanks.

Can anyone help me? This problem has been bothering me for a long time.

Hi @wenj1104,

Welcome to our community. :wave:

I saw that you also opened a GitHub Issue for this (hashicorp/consul#7312). The first part of the log message that you posted, cache.getCommonPorts: Fail to parse - port, is not present anywhere in Consul’s code base. This particular error is probably being generated by another application on the system.

Are you able to run Consul in debug mode, and provide a snippet of those logs? That might help to provide more information & assist in debugging the high CPU utilization.

Thank you.

1 Like