I’m using Consul 1.10.0. I’m not using ACLs.
I think I identified the issue. This happens if the secondary datacenter has a config entry separate from the one stored in the primary data center. In this case updates to that entry will not get replicated to the secondary. Consul will forward GET requests to the primary only if that config entry does not exist in the secondary data center but in this case it does so GET requests do not get forwarded to the primary and I end up seeing a stale value in the secondary datacenter.
It might be related to this Github issue Consul configs on secondary dc for federated datacenters - 1.9.3 · Issue #10356 · hashicorp/consul · GitHub.
I think this could happen if a config entry is written to the secondary data center before it joins the primary. Or maybe it happens if the primary goes down temporarily and config entries get written to the secondary datacenter before the primary recovers. Please let me know if this sounds possible.
This is hard to fix because PUT/DELETE requests DO get forwarded to the primary datacenter so it’s not possible to delete values from the secondary datacenter directly. I’ve had to take down the primary datacenter and then delete the values from the secondary in order to get the DELETE requests sent to the secondary datacenter.
Does any of this sound familiar? Am on the right track with this? I’m looking for a better way to handle this.