All Consul pods are consuming 100% of its allocated cpu (500m - 1000m). Vault agent container is top
This is a description of container:
Name: consul-server-0
Namespace: hashicorp
Priority: 0
Service Account: consul-server
Node: 10.42.1.21/10.42.1.21
Start Time: Sat, 06 Jul 2024 08:54:59 -0300
Labels: app=consul
apps.kubernetes.io/pod-index=0
chart=consul-helm
component=server
controller-revision-hash=consul-server-649b4d65bf
hasDNS=true
release=consul
statefulset.kubernetes.io/pod-name=consul-server-0
Annotations: consul.hashicorp.com/config-checksum: b080f4acc849ea969b6212d2934d342c25a32503a6d0573c21e6357b54ca1274
consul.hashicorp.com/connect-inject: false
consul.hashicorp.com/mesh-inject: false
vault.hashicorp.com/agent-inject: true
vault.hashicorp.com/agent-inject-secret-bootstrap-token-config.hcl: consul-kv/data/secrets/acl/bootstrap-token
vault.hashicorp.com/agent-inject-secret-gossip.txt: consul-kv/data/secrets/gossip
vault.hashicorp.com/agent-inject-secret-serverca.crt: pki_consul/cert/ca
vault.hashicorp.com/agent-inject-secret-servercert.crt: pki_consul/issue/consul-server
vault.hashicorp.com/agent-inject-secret-servercert.key: pki_consul/issue/consul-server
vault.hashicorp.com/agent-inject-status: injected
vault.hashicorp.com/agent-inject-template-bootstrap-token-config.hcl:
{{- with secret "consul-kv/data/secrets/acl/bootstrap-token" -}}
acl { tokens { initial_management = "{{- .Data.data.token -}}" }}
{{- end -}}
vault.hashicorp.com/agent-inject-template-gossip.txt:
{{- with secret "consul-kv/data/secrets/gossip" -}}
{{- .Data.data.key -}}
{{- end -}}
vault.hashicorp.com/agent-inject-template-serverca.crt:
{{- with secret "pki_consul/cert/ca" -}}
{{- .Data.certificate -}}
{{- end -}}
vault.hashicorp.com/agent-inject-template-servercert.crt:
{{- with secret "pki_consul/issue/consul-server" "common_name=server.vinhedo.consul"
"alt_names=localhost,consul-server,*.consul-server,*.consul-server.hashicorp,consul-server.hashicorp,*.consul-server.hashicorp.svc,consul-...
{{- .Data.certificate -}}
{{- if .Data.ca_chain -}}
{{- $lastintermediatecertindex := len .Data.ca_chain | subtract 1 -}}
{{ range $index, $cacert := .Data.ca_chain }}
{{ if (lt $index $lastintermediatecertindex) }}
{{ $cacert }}
{{ end }}
{{ end }}
{{- end -}}
{{- end -}}
vault.hashicorp.com/agent-inject-template-servercert.key:
{{- with secret "pki_consul/issue/consul-server" "common_name=server.vinhedo.consul"
"alt_names=localhost,consul-server,*.consul-server,*.consul-server.hashicorp,consul-server.hashicorp,*.consul-server.hashicorp.svc,consul-...
{{- .Data.private_key -}}
{{- end -}}
vault.hashicorp.com/role: consul-server
Status: Running
IP: 10.244.6.83
IPs:
IP: 10.244.6.83
Controlled By: StatefulSet/consul-server
Init Containers:
locality-init:
Container ID: cri-o://7a31b360e5809653f049e0618771868066c200300387d96654e5fc0b59beb28e
Image: hashicorp/consul-k8s-control-plane:1.5.0
Image ID: d3d37fdda10392698d48574b223339808bfb166c4b2b5fe9b6121f07c8ad90e0
Port: <none>
Host Port: <none>
SeccompProfile: RuntimeDefault
Command:
/bin/sh
-ec
exec consul-k8s-control-plane fetch-server-region -node-name "$NODE_NAME" -output-file /consul/extra-config/locality.json
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sat, 06 Jul 2024 08:55:02 -0300
Finished: Sat, 06 Jul 2024 08:55:02 -0300
Ready: True
Restart Count: 0
Environment:
NODE_NAME: (v1:spec.nodeName)
Mounts:
/consul/extra-config from extra-config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-d8h9v (ro)
/vault/secrets from vault-secrets (rw)
vault-agent-init:
Container ID: cri-o://b70009efa0423d8a92b4bd2574cf71cfff109f3cd9e9503419124ecab3b42dce
Image: hashicorp/vault:1.17.0
Image ID: 6bca409706dabae7df50f15e5f9dd0152a87a7646a61cfc0ce42fbbe53077950
Port: <none>
Host Port: <none>
Command:
/bin/sh
-ec
Args:
echo ${VAULT_CONFIG?} | base64 -d > /home/vault/config.json && vault agent -config=/home/vault/config.json
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sat, 06 Jul 2024 08:55:02 -0300
Finished: Sat, 06 Jul 2024 08:55:03 -0300
Ready: True
Restart Count: 0
Limits:
cpu: 500m
memory: 128Mi
Requests:
cpu: 250m
memory: 64Mi
Environment:
NAMESPACE: hashicorp (v1:metadata.namespace)
HOST_IP: (v1:status.hostIP)
POD_IP: (v1:status.podIP)
VAULT_LOG_LEVEL: info
VAULT_LOG_FORMAT: standard
Mounts:
/home/vault from home-init (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-d8h9v (ro)
/vault/secrets from vault-secrets (rw)
Containers:
consul:
Container ID: cri-o://e35c5a9b7963eb81165e1b4faee1eff2658ddda0b6d6df089f2a70c7038be32e
Image: hashicorp/consul:1.19.0
Image ID: bb7114bcaf5225329144303e67841fd4613bd9328d5f0d516db08799b23a7f2a
Ports: 8501/TCP, 8502/TCP, 8301/TCP, 8301/UDP, 8302/TCP, 8302/UDP, 8300/TCP, 8600/TCP, 8600/UDP
Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/UDP, 0/TCP, 0/UDP, 0/TCP, 0/TCP, 0/UDP
SeccompProfile: RuntimeDefault
Command:
/bin/sh
-ec
GOSSIP_KEY=`cat /vault/secrets/gossip.txt`
cp /consul/tmp/extra-config/extra-from-values.json /consul/extra-config/extra-from-values.json
[ -n "${HOST_IP}" ] && sed -Ei "s|HOST_IP|${HOST_IP?}|g" /consul/extra-config/extra-from-values.json
[ -n "${POD_IP}" ] && sed -Ei "s|POD_IP|${POD_IP?}|g" /consul/extra-config/extra-from-values.json
[ -n "${HOSTNAME}" ] && sed -Ei "s|HOSTNAME|${HOSTNAME?}|g" /consul/extra-config/extra-from-values.json
exec /usr/local/bin/docker-entrypoint.sh consul agent \
-advertise="${ADVERTISE_IP}" \
-config-dir=/consul/config \
-encrypt="${GOSSIP_KEY}" \
-config-file=/vault/secrets/bootstrap-token-config.hcl \
-config-dir=/consul/extra-config \
State: Running
Started: Sat, 06 Jul 2024 08:55:04 -0300
Ready: True
Restart Count: 0
Limits:
cpu: 100m
memory: 200Mi
Requests:
cpu: 100m
memory: 200Mi
Readiness: exec [/bin/sh -ec curl -k \
https://127.0.0.1:8501/v1/status/leader \
2>/dev/null | grep -E '".+"'
] delay=5s timeout=5s period=3s #success=1 #failure=2
Environment:
ADVERTISE_IP: (v1:status.podIP)
HOST_IP: (v1:status.hostIP)
POD_IP: (v1:status.podIP)
CONSUL_DISABLE_PERM_MGMT: true
CONSUL_HTTP_ADDR: https://localhost:8501
CONSUL_CACERT: /vault/secrets/serverca.crt
Mounts:
/consul/config from config (rw)
/consul/data from data-hashicorp (rw)
/consul/extra-config from extra-config (rw)
/consul/tmp/extra-config from tmp-extra-config (rw)
/tmp from tmp (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-d8h9v (ro)
/vault/secrets from vault-secrets (rw)
vault-agent:
Container ID: cri-o://7e77d5e5ad75e5eff57fb13e9168e8dd35548c482b8cdec4834c23c8926f174c
Image: hashicorp/vault:1.17.0
Image ID: 6bca409706dabae7df50f15e5f9dd0152a87a7646a61cfc0ce42fbbe53077950
Port: <none>
Host Port: <none>
Command:
/bin/sh
-ec
Args:
echo ${VAULT_CONFIG?} | base64 -d > /home/vault/config.json && vault agent -config=/home/vault/config.json
State: Running
Started: Sat, 06 Jul 2024 08:55:04 -0300
Ready: True
Restart Count: 0
Limits:
cpu: 500m
memory: 128Mi
Requests:
cpu: 250m
memory: 64Mi
Environment:
NAMESPACE: hashicorp (v1:metadata.namespace)
HOST_IP: (v1:status.hostIP)
POD_IP: (v1:status.podIP)
VAULT_LOG_LEVEL: info
VAULT_LOG_FORMAT: standard
VAULT_CONFIG: eyJhdXRvX2F1dGgiOnsibWV0aG9kIjp7InR5cGUiOiJrdWJlcm5ldGVzIiwibW91bnRfcGF0aCI6ImF1dGgva3ViZXJuZXRlcyIsImNvbmZpZyI6eyJyb2xlIjoiY29uc3VsLXNlcnZlciIsInRva2VuX3BhdGgiOiIvdmFyL3J1bi9zZWNyZXRzL2t1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvdG9rZW4ifX0sInNpbmsiOlt7InR5cGUiOiJmaWxlIiwiY29uZmlnIjp7InBhdGgiOiIvaG9tZS92YXVsdC8udmF1bHQtdG9rZW4ifX1dfSwiZXhpdF9hZnRlcl9hdXRoIjpmYWxzZSwicGlkX2ZpbGUiOiIvaG9tZS92YXVsdC8ucGlkIiwidmF1bHQiOnsiYWRkcmVzcyI6Imh0dHBzOi8vdmF1bHQudmluaGVkby5rdWdlbGJpdC53b3JrIn0sInRlbXBsYXRlIjpbeyJkZXN0aW5hdGlvbiI6Ii92YXVsdC9zZWNyZXRzL3NlcnZlcmNlcnQuY3J0IiwiY29udGVudHMiOiJ7ey0gd2l0aCBzZWNyZXQgXCJwa2lfY29uc3VsL2lzc3VlL2NvbnN1bC1zZXJ2ZXJcIiBcImNvbW1vbl9uYW1lPXNlcnZlci52aW5oZWRvLmNvbnN1bFwiXG5cImFsdF9uYW1lcz1sb2NhbGhvc3QsY29uc3VsLXNlcnZlciwqLmNvbnN1bC1zZXJ2ZXIsKi5jb25zdWwtc2VydmVyLmhhc2hpY29ycCxjb25zdWwtc2VydmVyLmhhc2hpY29ycCwqLmNvbnN1bC1zZXJ2ZXIuaGFzaGljb3JwLnN2Yyxjb25zdWwtc2VydmVyLmhhc2hpY29ycC5zdmMsKi5zZXJ2ZXIudmluaGVkby5jb25zdWwsY29uc3VsLnZpbmhlZG8ua3VnZWxiaXQud29ya1wiIFwiaXBfc2Fucz0xMjcuMC4wLjFcIiAtfX1cbnt7LSAuRGF0YS5jZXJ0aWZpY2F0ZSAtfX1cbnt7LSBpZiAuRGF0YS5jYV9jaGFpbiAtfX1cbnt7LSAkbGFzdGludGVybWVkaWF0ZWNlcnRpbmRleCA6PSBsZW4gLkRhdGEuY2FfY2hhaW4gfCBzdWJ0cmFjdCAxIC19fVxue3sgcmFuZ2UgJGluZGV4LCAkY2FjZXJ0IDo9IC5EYXRhLmNhX2NoYWluIH19XG57eyBpZiAobHQgJGluZGV4ICRsYXN0aW50ZXJtZWRpYXRlY2VydGluZGV4KSB9fVxue3sgJGNhY2VydCB9fVxue3sgZW5kIH19XG57eyBlbmQgfX1cbnt7LSBlbmQgLX19XG57ey0gZW5kIC19fVxuIiwibGVmdF9kZWxpbWl0ZXIiOiJ7eyIsInJpZ2h0X2RlbGltaXRlciI6In19In0seyJkZXN0aW5hdGlvbiI6Ii92YXVsdC9zZWNyZXRzL2Jvb3RzdHJhcC10b2tlbi1jb25maWcuaGNsIiwiY29udGVudHMiOiJ7ey0gd2l0aCBzZWNyZXQgXCJjb25zdWwta3YvZGF0YS9zZWNyZXRzL2FjbC9ib290c3RyYXAtdG9rZW5cIiAtfX1cbmFjbCB7IHRva2VucyB7IGluaXRpYWxfbWFuYWdlbWVudCA9IFwie3stIC5EYXRhLmRhdGEudG9rZW4gLX19XCIgfX1cbnt7LSBlbmQgLX19XG4iLCJsZWZ0X2RlbGltaXRlciI6Int7IiwicmlnaHRfZGVsaW1pdGVyIjoifX0ifSx7ImRlc3RpbmF0aW9uIjoiL3ZhdWx0L3NlY3JldHMvc2VydmVyY2EuY3J0IiwiY29udGVudHMiOiJ7ey0gd2l0aCBzZWNyZXQgXCJwa2lfY29uc3VsL2NlcnQvY2FcIiAtfX1cbnt7LSAuRGF0YS5jZXJ0aWZpY2F0ZSAtfX1cbnt7LSBlbmQgLX19XG4iLCJsZWZ0X2RlbGltaXRlciI6Int7IiwicmlnaHRfZGVsaW1pdGVyIjoifX0ifSx7ImRlc3RpbmF0aW9uIjoiL3ZhdWx0L3NlY3JldHMvc2VydmVyY2VydC5rZXkiLCJjb250ZW50cyI6Int7LSB3aXRoIHNlY3JldCBcInBraV9jb25zdWwvaXNzdWUvY29uc3VsLXNlcnZlclwiIFwiY29tbW9uX25hbWU9c2VydmVyLnZpbmhlZG8uY29uc3VsXCJcblwiYWx0X25hbWVzPWxvY2FsaG9zdCxjb25zdWwtc2VydmVyLCouY29uc3VsLXNlcnZlciwqLmNvbnN1bC1zZXJ2ZXIuaGFzaGljb3JwLGNvbnN1bC1zZXJ2ZXIuaGFzaGljb3JwLCouY29uc3VsLXNlcnZlci5oYXNoaWNvcnAuc3ZjLGNvbnN1bC1zZXJ2ZXIuaGFzaGljb3JwLnN2YywqLnNlcnZlci52aW5oZWRvLmNvbnN1bCxjb25zdWwudmluaGVkby5rdWdlbGJpdC53b3JrXCIgXCJpcF9zYW5zPTEyNy4wLjAuMVwiIC19fVxue3stIC5EYXRhLnByaXZhdGVfa2V5IC19fVxue3stIGVuZCAtfX1cbiIsImxlZnRfZGVsaW1pdGVyIjoie3siLCJyaWdodF9kZWxpbWl0ZXIiOiJ9fSJ9LHsiZGVzdGluYXRpb24iOiIvdmF1bHQvc2VjcmV0cy9nb3NzaXAudHh0IiwiY29udGVudHMiOiJ7ey0gd2l0aCBzZWNyZXQgXCJjb25zdWwta3YvZGF0YS9zZWNyZXRzL2dvc3NpcFwiIC19fVxue3stIC5EYXRhLmRhdGEua2V5IC19fVxue3stIGVuZCAtfX1cbiIsImxlZnRfZGVsaW1pdGVyIjoie3siLCJyaWdodF9kZWxpbWl0ZXIiOiJ9fSJ9XSwidGVtcGxhdGVfY29uZmlnIjp7ImV4aXRfb25fcmV0cnlfZmFpbHVyZSI6dHJ1ZX19
Mounts:
/home/vault from home-sidecar (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-d8h9v (ro)
/vault/secrets from vault-secrets (rw)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
data-hashicorp:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: data-hashicorp-consul-server-0
ReadOnly: false
tmp:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: consul-server-config
Optional: false
extra-config:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
tmp-extra-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: consul-server-tmp-extra-config
Optional: false
kube-api-access-d8h9v:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
home-init:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
home-sidecar:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
vault-secrets:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
QoS Class: Burstable
Node-Selectors: kubernetes.io/arch=amd64
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 49m default-scheduler Successfully assigned hashicorp/consul-server-0 to 10.42.1.21
Warning FailedMount 49m kubelet MountVolume.SetUp failed for volume "tmp-extra-config" : failed to sync configmap cache: timed out waiting for the condition
Normal Pulled 49m kubelet Container image "hashicorp/consul-k8s-control-plane:1.5.0" already present on machine
Normal Created 49m kubelet Created container locality-init
Normal Started 49m kubelet Started container locality-init
Normal Pulled 49m kubelet Container image "hashicorp/vault:1.17.0" already present on machine
Normal Created 49m kubelet Created container vault-agent-init
Normal Started 49m kubelet Started container vault-agent-init
Normal Pulled 49m kubelet Container image "hashicorp/consul:1.19.0" already present on machine
Normal Created 49m kubelet Created container consul
Normal Started 49m kubelet Started container consul
Normal Pulled 49m kubelet Container image "hashicorp/vault:1.17.0" already present on machine
Normal Created 49m kubelet Created container vault-agent
Normal Started 49m kubelet Started container vault-agent
Warning Unhealthy 49m kubelet Readiness probe failed: