Consul keeps high CPU utilization

Hi there,

We deployed neuvector in our k8s cluster. And the consul process is in all the neuvector pod. So I think there should be a consul cluster, right?

Then, we found the consul process consumed so much CPU.

And we found so many error logs. Following is part of them.

2020-03-18T23:11:06.533|ERRO|CTL|cache.getCommonPorts: Fail to parse - port=
2020-03-18T23:11:06.533|ERRO|CTL|cache.getCommonPorts: Fail to parse - port=
2020-03-18T23:11:06.533|ERRO|CTL|cache.getCommonPorts: Fail to parse - port=
2020-03-18T23:11:06.533|ERRO|CTL|cache.getCommonPorts: Fail to parse - port=
2020-03-18T23:11:06.533|ERRO|CTL|cache.getCommonPorts: Fail to parse - port=
    2020/03/18 23:12:48 [INFO] consul.fsm: snapshot created in 38.007µs
    2020/03/18 23:12:48 [INFO] raft: Starting snapshot up to 122048314
    2020/03/18 23:12:48 [INFO] snapshot: Creating new snapshot at /tmp/neuvector/raft/snapshots/2-122048314-1584573168213.tmp
    2020/03/18 23:12:48 [INFO] snapshot: reaping snapshot /tmp/neuvector/raft/snapshots/2-122015057-1584570214964
    2020/03/18 23:12:48 [INFO] raft: Compacting logs from 122021420 to 122038074
    2020/03/18 23:12:48 [INFO] raft: Snapshot to 122048314 complete
2020-03-18T23:22:24.872|INFO|CTL|cache.controllerUpdate: Add or update controller - controller={CLUSDevice:{ID:c60912fed4b4c222cde27c55b6105f2d814667fb6e296a70b49908c71cc8dda4 Name:k8s_neuvector-controller-pod_neuvector-controller-pod-54c4ccc89-tnd8d_default_9e9a2694-06e4-11ea-a61a-525400d4fd47_1 SelfHostname:neuvector-controller-pod-54c4ccc89-tnd8d HostName:tc-10-189-169-145 HostID:tc-10-189-169-145:5LDF:4OY6:XBPD:IIIZ:HL2Q:X73I:ZAHD:5DX2:ZB2O:C77K:AGW7:QB6O Domain:default NetworkMode:container:0fd4cc3eed9249289795d5799c8f64617f235e79585dae3bc723eb7ca484b93b PidMode: Ver:v2.5.1 Labels:map[annotation.io.kubernetes.container.hash:2be7c349 annotation.io.kubernetes.container.preStopHandler:{"exec":{"command":["/usr/local/bin/consul","leave"]}} annotation.io.kubernetes.container.restartCount:1 annotation.io.kubernetes.container.terminationMessagePath:/dev/termination-log annotation.io.kubernetes.container.terminationMessagePolicy:File annotation.io.kubernetes.pod.terminationGracePeriod:60 io.kubernetes.container.logpath:/var/log/pods/9e9a2694-06e4-11ea-a61a-525400d4fd47/neuvector-controller-pod/1.log io.kubernetes.container.name:neuvector-controller-pod io.kubernetes.docker.type:container io.kubernetes.pod.name:neuvector-controller-pod-54c4ccc89-tnd8d io.kubernetes.pod.namespace:default io.kubernetes.pod.uid:9e9a2694-06e4-11ea-a61a-525400d4fd47 io.kubernetes.sandbox.id:0fd4cc3eed9249289795d5799c8f64617f235e79585dae3bc723eb7ca484b93b neuvector.image:neuvector/controller neuvector.role:controller] CreatedAt:2020-01-02 05:49:56.751504169 +0000 UTC StartedAt:2020-01-02 05:49:56.926307661 +0000 UTC JoinedAt:2020-01-02 05:50:02.386014827 +0000 UTC MemoryLimit:0 CPUs: ClusterIP:172.16.56.41 RPCServerPort:18400 Pid:29550 Ifaces:map[eth0:[{IPNet:{IP:172.16.56.41 Mask:ffffff00} Gateway: Scope:global NetworkID: NetworkName:}]]} Leader:false OrchConnStatus:connected OrchConnLastError:}

Can someone give me some advice or tell me how to investigate it?

By the way, in the neuvector pods, the version of consul is v1.5.1.

# consul version
Consul v1.5.1
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

And the status seems good.

# consul members
Node           Address              Status  Type    Build  Protocol  DC         Segment
172.16.15.11   172.16.15.11:18301   alive   server  1.5.1  2         neuvector  <all>
172.16.33.8    172.16.33.8:18301    alive   server  1.5.1  2         neuvector  <all>
172.16.56.41   172.16.56.41:18301   alive   server  1.5.1  2         neuvector  <all>
172.16.10.13   172.16.10.13:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.11.10   172.16.11.10:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.12.11   172.16.12.11:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.13.7    172.16.13.7:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.14.8    172.16.14.8:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.15.12   172.16.15.12:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.16.8    172.16.16.8:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.17.9    172.16.17.9:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.18.14   172.16.18.14:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.19.12   172.16.19.12:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.20.12   172.16.20.12:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.21.9    172.16.21.9:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.22.8    172.16.22.8:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.23.11   172.16.23.11:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.24.9    172.16.24.9:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.25.8    172.16.25.8:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.26.10   172.16.26.10:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.27.8    172.16.27.8:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.28.10   172.16.28.10:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.29.11   172.16.29.11:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.3.11    172.16.3.11:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.30.10   172.16.30.10:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.31.10   172.16.31.10:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.32.9    172.16.32.9:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.33.9    172.16.33.9:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.34.14   172.16.34.14:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.35.9    172.16.35.9:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.36.8    172.16.36.8:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.37.17   172.16.37.17:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.38.9    172.16.38.9:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.39.18   172.16.39.18:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.4.11    172.16.4.11:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.40.9    172.16.40.9:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.41.18   172.16.41.18:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.42.11   172.16.42.11:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.43.185  172.16.43.185:18301  alive   client  1.5.1  2         neuvector  <default>
172.16.44.57   172.16.44.57:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.45.15   172.16.45.15:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.46.55   172.16.46.55:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.47.103  172.16.47.103:18301  alive   client  1.5.1  2         neuvector  <default>
172.16.48.48   172.16.48.48:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.49.63   172.16.49.63:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.5.7     172.16.5.7:18301     alive   client  1.5.1  2         neuvector  <default>
172.16.50.49   172.16.50.49:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.51.46   172.16.51.46:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.52.44   172.16.52.44:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.53.48   172.16.53.48:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.54.57   172.16.54.57:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.55.50   172.16.55.50:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.56.40   172.16.56.40:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.57.8    172.16.57.8:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.58.33   172.16.58.33:18301   alive   client  1.5.1  2         neuvector  <default>
172.16.59.9    172.16.59.9:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.6.15    172.16.6.15:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.60.4    172.16.60.4:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.61.2    172.16.61.2:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.62.4    172.16.62.4:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.63.4    172.16.63.4:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.64.4    172.16.64.4:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.65.2    172.16.65.2:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.66.4    172.16.66.4:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.67.2    172.16.67.2:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.68.3    172.16.68.3:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.69.3    172.16.69.3:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.7.8     172.16.7.8:18301     alive   client  1.5.1  2         neuvector  <default>
172.16.70.4    172.16.70.4:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.71.4    172.16.71.4:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.72.3    172.16.72.3:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.73.2    172.16.73.2:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.74.2    172.16.74.2:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.8.10    172.16.8.10:18301    alive   client  1.5.1  2         neuvector  <default>
172.16.9.12    172.16.9.12:18301    alive   client  1.5.1  2         neuvector  <default>

If you need more information, please let me know. Thanks.

Can anyone help me? This problem has been bothering me for a long time.