Error performing anti-entropy sync of federation state

Hello all,

I have a DC running and recently I noticed the following error message and I can’t find anything about it or can’t quite figure out what to do.

can give me a nudge in one direction …

Thanks and greetings

consul v1.16.0

2023-07-12T21:10:33.278+0200 [INFO]  agent: Caught: signal=terminated
2023-07-12T21:10:33.278+0200 [INFO]  agent: Graceful shutdown disabled. Exiting
2023-07-12T21:10:33.278+0200 [INFO]  agent: Requesting shutdown
2023-07-12T21:10:33.278+0200 [INFO]  agent.server: shutting down server
2023-07-12T21:10:33.278+0200 [INFO]  agent.leader: stopping routine: routine="metrics for streaming peering resources"
2023-07-12T21:10:33.278+0200 [INFO]  agent.leader: stopping routine: routine="CA root expiration metric"
2023-07-12T21:10:33.278+0200 [INFO]  agent.leader: stopping routine: routine="CA signing expiration metric"
2023-07-12T21:10:33.278+0200 [INFO]  agent.leader: stopping routine: routine="config entry controllers"
2023-07-12T21:10:33.278+0200 [INFO]  agent.leader: stopping routine: routine="acl token reaping"
2023-07-12T21:10:33.278+0200 [INFO]  agent.leader: stopping routine: routine="federation state anti-entropy"
2023-07-12T21:10:33.278+0200 [INFO]  agent.leader: stopping routine: routine="federation state pruning"
2023-07-12T21:10:33.278+0200 [INFO]  agent.leader: stopping routine: routine="streaming peering resources"
2023-07-12T21:10:33.278+0200 [INFO]  agent.leader: stopping routine: routine="peering deferred deletion"
2023-07-12T21:10:33.278+0200 [INFO]  agent.leader: stopping routine: routine="intermediate cert renew watch"
2023-07-12T21:10:33.278+0200 [INFO]  agent.leader: stopping routine: routine="CA root pruning"
2023-07-12T21:10:33.278+0200 [WARN]  agent.server.serf.lan: serf: Shutdown without a Leave
2023-07-12T21:10:33.278+0200 [INFO]  agent.leader: stopped routine: routine="config entry controllers"
2023-07-12T21:10:33.278+0200 [INFO]  agent.leader: stopped routine: routine="acl token reaping"
2023-07-12T21:10:33.278+0200 [INFO]  agent.leader: stopped routine: routine="federation state pruning"
2023-07-12T21:10:33.278+0200 [INFO]  agent.leader: stopped routine: routine="streaming peering resources"
2023-07-12T21:10:33.278+0200 [INFO]  agent.leader: stopped routine: routine="peering deferred deletion"
2023-07-12T21:10:33.278+0200 [INFO]  agent.leader: stopped routine: routine="intermediate cert renew watch"
2023-07-12T21:10:33.278+0200 [INFO]  agent.leader: stopped routine: routine="CA root pruning"
2023-07-12T21:10:33.279+0200 [INFO]  agent.leader: stopping routine: routine="federation state anti-entropy"
2023-07-12T21:10:33.279+0200 [INFO]  agent.leader: stopping routine: routine="metrics for streaming peering resources"
2023-07-12T21:10:33.279+0200 [INFO]  agent.leader: stopping routine: routine="CA root expiration metric"
2023-07-12T21:10:33.279+0200 [INFO]  agent.leader: stopping routine: routine="CA signing expiration metric"
2023-07-12T21:10:33.279+0200 [INFO]  agent.server.autopilot: reconciliation now disabled
2023-07-12T21:10:33.282+0200 [ERROR] agent.server: error performing anti-entropy sync of federation state: error="context canceled"
2023-07-12T21:10:33.282+0200 [INFO]  agent.leader: stopped routine: routine="federation state anti-entropy"
2023-07-12T21:10:33.282+0200 [INFO]  agent.server: shutting down Connect CA roots monitor
2023-07-12T21:10:33.282+0200 [INFO]  agent.server.peering_metrics: stopping peering metrics
2023-07-12T21:10:33.282+0200 [INFO]  agent.leader: stopped routine: routine="metrics for streaming peering resources"
2023-07-12T21:10:33.282+0200 [INFO]  agent.leader: stopped routine: routine="CA root expiration metric"
2023-07-12T21:10:33.282+0200 [INFO]  agent.leader: stopped routine: routine="CA signing expiration metric"
2023-07-12T21:10:33.282+0200 [WARN]  agent.controller-runtime: error received from watch: managed_type=internal.v1.tombstone error="rpc error: code = Canceled desc = context canceled"
2023-07-12T21:10:33.296+0200 [WARN]  agent.server.serf.wan: serf: Shutdown without a Leave
2023-07-12T21:10:33.309+0200 [INFO]  agent.router.manager: shutting down
2023-07-12T21:10:33.309+0200 [INFO]  agent.router.manager: shutting down
2023-07-12T21:10:33.310+0200 [INFO]  agent: consul server down
2023-07-12T21:10:33.310+0200 [INFO]  agent: shutdown complete
2023-07-12T21:10:33.310+0200 [INFO]  agent: Stopping server: protocol=DNS address=0.0.0.0:8600 network=tcp
2023-07-12T21:10:33.310+0200 [INFO]  agent: Stopping server: protocol=DNS address=0.0.0.0:8600 network=udp
2023-07-12T21:10:33.310+0200 [INFO]  agent: Stopping server: address=[::]:8500 network=tcp protocol=http

Remote side ended the monitor! This usually means that the
remote side has exited or crashed.

The error doesn’t matter. It just means “Consul was in the middle of doing stuff but it cancelled that processing when it noticed a shutdown was requested”.

Thank you for the quick answer.

What I just do not quite understand is why this happens constantly … ?
Because of this the consul monitor command is always aborted and
that is a bit annoying when troubleshooting.

Er… what?

Of course the consul monitor command is going to end, if you shut down the Consul server it was monitoring - because the server is no longer running for the monitor process to talk to!

The error message you mentioned is a mere side-effect of the Consul server being shut down by some other trigger, not the cause.

I do not shut down the server …

I think it might have something to do with this. This is the output of one of the agents.

2023-07-13T09:54:59.280+0200 [WARN]  agent.client: Retrying RPC to server: method=KVS.List server=ip:8300 error="rpc error making call: EOF"
2023-07-13T09:54:59.284+0200 [WARN]  agent.client: Retrying RPC to server: method=ConnectCA.Roots server=ip.73:8300 error="rpc error making call: EOF"
2023-07-13T09:54:59.977+0200 [INFO]  agent.client.serf.lan: serf: EventMemberUpdate: consul.domain
2023-07-13T09:54:59.977+0200 [INFO]  agent.client: updating server: server="consul.domain (Addr: tcp/ip:8300) (DC: consul-dc1)"
2023-07-13T09:55:08.488+0200 [ERROR] agent.client: RPC failed to server: method=KVS.Get server=ip:8300 error="rpc error making call: i/o deadline reached"
2023-07-13T09:55:08.488+0200 [ERROR] agent.http: Request error: method=GET url="/v1/kv/PostgreSQL/postgreSQL-13/leader?consistent=&index=11321904&wait=1000ms" from=127.0.0.1:57578 error="rpc error making call: i/o deadline reached"
2023-07-13T09:55:09.285+0200 [ERROR] agent.client: RPC failed to server: method=KVS.List server=ip:8300 error="rpc error making call: No cluster leader"
2023-07-13T09:55:09.285+0200 [INFO]  agent.http: Request cancelled: method=GET url=/v1/kv/PostgreSQL/postgreSQL-13/?recurse=1 from=127.0.0.1:41788 error="rpc error making call: No cluster leader"
2023-07-13T09:55:09.976+0200 [INFO]  agent.client: New leader elected: payload=consul.domain

Something told the server to shut down:

That appears to just be more consequences of a server being shut down.