Servers in secondary datacenter complains about failed CA operation set-root-config

Hello guys,

we have recently upgraded Consul in our test environment to version 1.11.2 and servers in secondary datacenter started to show following logs (every 10-15 mins):

Feb  3 23:13:56 oconsul1 consul[5031]: 2022-02-03T23:13:56.911+0100 [WARN]  agent.fsm: Failed to apply CA operation: operation=set-roots-config
Feb  3 23:24:12 oconsul1 consul[5031]: 2022-02-03T23:24:12.986+0100 [WARN]  agent.fsm: Failed to apply CA operation: operation=set-roots-config
Feb  3 23:34:20 oconsul1 consul[5031]: 2022-02-03T23:34:20.899+0100 [WARN]  agent.fsm: Failed to apply CA operation: operation=set-roots-config

Context: We have two Consul datacenters in WAN Federation setup with Mesh Gateways (we are unable to interconnect both environments on IP level). Everything was deployed by hand, no Kubernetes (or helm) was envolved.

Currently we use it mainly for testing Service Discovery, we have Connect enabled but we don’t use it except for Mesh Gateway federation. Token replication, Service discovery and general access cross datacenter works just fine.

I thought I might have some ACLs or Policies misconfigured but I had no luck fiddling with that.