Hello,
I am continuously getting below error in consul deployed in k8s v1.23.4.
Consul Version: 1.11.4
[ERROR] agent: Coordinate update error: error=“rpc error making call: stream closed”
Following tests are found good, No issues
command: consul members
consul operator raft list-peers
Leader is elected. No issues.
Any help is very much appreciated. For debugging please let me some steps.
The below error is found in the consul server logs.
==> Starting Consul agent…
Version: ‘1.11.4’
Node ID: ‘5bae968b-c936-9196-97d9-6cc0027c1358’
Node name: ‘consul-server-0’
Datacenter: ‘consul-dc’ (Segment: ‘’)
Server: true (Bootstrap: false)
Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: 8503, DNS: 8600)
Cluster Addr: 10.244.0.130 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
2022-10-14T08:36:37.042Z [WARN] agent: error getting server health from server: server=cortex-consul-server-1 error=“rpc error making call: stream closed”
2022-10-14T08:36:38.042Z [WARN] agent: error getting server health from server: server=cortex-consul-server-2 error=“context deadline exceeded”
2022-10-14T08:36:38.042Z [WARN] agent: error getting server health from server: server=cortex-consul-server-1 error=“context deadline exceeded”
2022-10-14T08:40:55.394Z [WARN] agent: grpc: addrConn.createTransport failed to connect to {cortex-dc-10.244.0.22:8300 0 cortex-consul-server-2.cortex-dc }. Err :connection error: desc = “transport: Error while dialing dial tcp ->10.244.0.22:8300: operation was canceled”. Reconnecting…
2022-10-14T08:41:44.089Z [INFO] agent.server.raft: aborting pipeline replication: peer=“{Voter 95b825e1-14af-db50-d1cb-57c920ef8f31 10.244.0.22:8300}”
2022-10-14T08:41:44.138Z [INFO] agent.server.raft: aborting pipeline replication: peer=“{Nonvoter fe51ef90-fc39-dd3a-3d97-9dc2b070c842 10.244.1.15:8300}”
2022-10-14T08:41:44.161Z [ERROR] agent.server.raft: failed to appendEntries to: peer=“{Voter 95b825e1-14af-db50-d1cb-57c920ef8f31 10.244.0.22:8300}” error=EOF
2022-10-14T08:41:44.203Z [ERROR] agent.server.raft: failed to appendEntries to: peer=“{Nonvoter fe51ef90-fc39-dd3a-3d97-9dc2b070c842 10.244.1.15:8300}” error=EOF