Error performing anti-entropy sync of federation state

Consul
1

Hello all,

I have a DC running and recently I noticed the following error message and I can’t find anything about it or can’t quite figure out what to do.

can give me a nudge in one direction …

Thanks and greetings

consul v1.16.0

2023-07-12T21:10:33.278+0200 [INFO]  agent: Caught: signal=terminated
2023-07-12T21:10:33.278+0200 [INFO]  agent: Graceful shutdown disabled. Exiting
2023-07-12T21:10:33.278+0200 [INFO]  agent: Requesting shutdown
2023-07-12T21:10:33.278+0200 [INFO]  agent.server: shutting down server
2023-07-12T21:10:33.278+0200 [INFO]  agent.leader: stopping routine: routine="metrics for streaming peering resources"
2023-07-12T21:10:33.278+0200 [INFO]  agent.leader: stopping routine: routine="CA root expiration metric"
2023-07-12T21:10:33.278+0200 [INFO]  agent.leader: stopping routine: routine="CA signing expiration metric"
2023-07-12T21:10:33.278+0200 [INFO]  agent.leader: stopping routine: routine="config entry controllers"
2023-07-12T21:10:33.278+0200 [INFO]  agent.leader: stopping routine: routine="acl token reaping"
2023-07-12T21:10:33.278+0200 [INFO]  agent.leader: stopping routine: routine="federation state anti-entropy"
2023-07-12T21:10:33.278+0200 [INFO]  agent.leader: stopping routine: routine="federation state pruning"
2023-07-12T21:10:33.278+0200 [INFO]  agent.leader: stopping routine: routine="streaming peering resources"
2023-07-12T21:10:33.278+0200 [INFO]  agent.leader: stopping routine: routine="peering deferred deletion"
2023-07-12T21:10:33.278+0200 [INFO]  agent.leader: stopping routine: routine="intermediate cert renew watch"
2023-07-12T21:10:33.278+0200 [INFO]  agent.leader: stopping routine: routine="CA root pruning"
2023-07-12T21:10:33.278+0200 [WARN]  agent.server.serf.lan: serf: Shutdown without a Leave
2023-07-12T21:10:33.278+0200 [INFO]  agent.leader: stopped routine: routine="config entry controllers"
2023-07-12T21:10:33.278+0200 [INFO]  agent.leader: stopped routine: routine="acl token reaping"
2023-07-12T21:10:33.278+0200 [INFO]  agent.leader: stopped routine: routine="federation state pruning"
2023-07-12T21:10:33.278+0200 [INFO]  agent.leader: stopped routine: routine="streaming peering resources"
2023-07-12T21:10:33.278+0200 [INFO]  agent.leader: stopped routine: routine="peering deferred deletion"
2023-07-12T21:10:33.278+0200 [INFO]  agent.leader: stopped routine: routine="intermediate cert renew watch"
2023-07-12T21:10:33.278+0200 [INFO]  agent.leader: stopped routine: routine="CA root pruning"
2023-07-12T21:10:33.279+0200 [INFO]  agent.leader: stopping routine: routine="federation state anti-entropy"
2023-07-12T21:10:33.279+0200 [INFO]  agent.leader: stopping routine: routine="metrics for streaming peering resources"
2023-07-12T21:10:33.279+0200 [INFO]  agent.leader: stopping routine: routine="CA root expiration metric"
2023-07-12T21:10:33.279+0200 [INFO]  agent.leader: stopping routine: routine="CA signing expiration metric"
2023-07-12T21:10:33.279+0200 [INFO]  agent.server.autopilot: reconciliation now disabled
2023-07-12T21:10:33.282+0200 [ERROR] agent.server: error performing anti-entropy sync of federation state: error="context canceled"
2023-07-12T21:10:33.282+0200 [INFO]  agent.leader: stopped routine: routine="federation state anti-entropy"
2023-07-12T21:10:33.282+0200 [INFO]  agent.server: shutting down Connect CA roots monitor
2023-07-12T21:10:33.282+0200 [INFO]  agent.server.peering_metrics: stopping peering metrics
2023-07-12T21:10:33.282+0200 [INFO]  agent.leader: stopped routine: routine="metrics for streaming peering resources"
2023-07-12T21:10:33.282+0200 [INFO]  agent.leader: stopped routine: routine="CA root expiration metric"
2023-07-12T21:10:33.282+0200 [INFO]  agent.leader: stopped routine: routine="CA signing expiration metric"
2023-07-12T21:10:33.282+0200 [WARN]  agent.controller-runtime: error received from watch: managed_type=internal.v1.tombstone error="rpc error: code = Canceled desc = context canceled"
2023-07-12T21:10:33.296+0200 [WARN]  agent.server.serf.wan: serf: Shutdown without a Leave
2023-07-12T21:10:33.309+0200 [INFO]  agent.router.manager: shutting down
2023-07-12T21:10:33.309+0200 [INFO]  agent.router.manager: shutting down
2023-07-12T21:10:33.310+0200 [INFO]  agent: consul server down
2023-07-12T21:10:33.310+0200 [INFO]  agent: shutdown complete
2023-07-12T21:10:33.310+0200 [INFO]  agent: Stopping server: protocol=DNS address=0.0.0.0:8600 network=tcp
2023-07-12T21:10:33.310+0200 [INFO]  agent: Stopping server: protocol=DNS address=0.0.0.0:8600 network=udp
2023-07-12T21:10:33.310+0200 [INFO]  agent: Stopping server: address=[::]:8500 network=tcp protocol=http

Remote side ended the monitor! This usually means that the
remote side has exited or crashed.
2

The error doesn’t matter. It just means “Consul was in the middle of doing stuff but it cancelled that processing when it noticed a shutdown was requested”.

3

Thank you for the quick answer.

What I just do not quite understand is why this happens constantly … ?
Because of this the consul monitor command is always aborted and
that is a bit annoying when troubleshooting.

4

Er… what?

Of course the consul monitor command is going to end, if you shut down the Consul server it was monitoring - because the server is no longer running for the monitor process to talk to!

The error message you mentioned is a mere side-effect of the Consul server being shut down by some other trigger, not the cause.

5

I do not shut down the server …

6

I think it might have something to do with this. This is the output of one of the agents.

2023-07-13T09:54:59.280+0200 [WARN]  agent.client: Retrying RPC to server: method=KVS.List server=ip:8300 error="rpc error making call: EOF"
2023-07-13T09:54:59.284+0200 [WARN]  agent.client: Retrying RPC to server: method=ConnectCA.Roots server=ip.73:8300 error="rpc error making call: EOF"
2023-07-13T09:54:59.977+0200 [INFO]  agent.client.serf.lan: serf: EventMemberUpdate: consul.domain
2023-07-13T09:54:59.977+0200 [INFO]  agent.client: updating server: server="consul.domain (Addr: tcp/ip:8300) (DC: consul-dc1)"
2023-07-13T09:55:08.488+0200 [ERROR] agent.client: RPC failed to server: method=KVS.Get server=ip:8300 error="rpc error making call: i/o deadline reached"
2023-07-13T09:55:08.488+0200 [ERROR] agent.http: Request error: method=GET url="/v1/kv/PostgreSQL/postgreSQL-13/leader?consistent=&index=11321904&wait=1000ms" from=127.0.0.1:57578 error="rpc error making call: i/o deadline reached"
2023-07-13T09:55:09.285+0200 [ERROR] agent.client: RPC failed to server: method=KVS.List server=ip:8300 error="rpc error making call: No cluster leader"
2023-07-13T09:55:09.285+0200 [INFO]  agent.http: Request cancelled: method=GET url=/v1/kv/PostgreSQL/postgreSQL-13/?recurse=1 from=127.0.0.1:41788 error="rpc error making call: No cluster leader"
2023-07-13T09:55:09.976+0200 [INFO]  agent.client: New leader elected: payload=consul.domain
7

Something told the server to shut down:

That appears to just be more consequences of a server being shut down.