Consul server going down with the error agent: syncing node info failed.: error="timed out enqueuing operation",
we have recently upgraded our kubernetes cluster from 1.23 to 1.27 and intermittently the consul server getting stopped. k8 version : 1.27, Consul helm chart :0.49.2 and app version is 1.13.4
Consul config is as follows
global:
name: consul
datacenter: dc1
image: hashicorp/consul:1.13.4
ui:
enabled: true
service:
enabled: true
type: ClusterIP
connectInject:
enabled: true
transparentProxy:
defaultEnabled: false
sidecarProxy:
resources:
requests:
memory: 500Mi
cpu: 500m
limits:
memory: 1000Mi
cpu: 500m
controller:
enabled: true
prometheus:
enabled: false
grafana:
enabled: false
securityContext:
runAsNonRoot: true
runAsGroup: 1000
runAsUser: 100
fsGroup: 1000
readinessProbe:
# NOTE(mitchellh): when our HTTP status endpoints support the
# proper status codes, we should switch to that. This is temporary.
exec:
command:
- “/bin/sh”
- “-ec”
- |
{{- if .Values.global.tls.enabled }}
curl
–cacert /consul/tls/ca/tls.crt
https://127.0.0.1:8501/v1/status/leader
{{- else }}
curl http://127.0.0.1:8500/v1/status/leader
{{- end }}
2>/dev/null | grep -E ‘“.+”’
failureThreshold: 5
initialDelaySeconds: 60
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5