I have two Consul data centers joined in a WAN. I am trying to create a service-resolver in the non-primary datacenter but noticed the config gets created in the primary data center instead.
Is this expected? GET requests seem to read from the correct data center but all PUT requests write to the primary one instead of the one I am sending the request to. Adding a ?dc= argument to my request doesn’t help either.
This is expected behavior. Config Entries are global in Consul’s data model. Read and write requests for config entries will be transparently forwarded to the primary datacenter regardless of what datacenter the request was sent to or what the
dc= query params value was.
Read requests are not getting routed to the primary data center though. I can only see new config entries if i send a GET request to the primary data center. GET requests to secondary data centers return nothing.
I saw you also asked this question at hashicorp-consul/Lobby - Gitter, and I replied with a few questions.
The fact that the read request does not return the entry sounds like there’s an issue with replication from the primary to secondary DCs. Which version of Consul are you using? Can you share the ACL policy that you have associated with the replication token that’s used in the secondary DC?
I’m using Consul 1.10.0. I’m not using ACLs.
I think I identified the issue. This happens if the secondary datacenter has a config entry separate from the one stored in the primary data center. In this case updates to that entry will not get replicated to the secondary. Consul will forward GET requests to the primary only if that config entry does not exist in the secondary data center but in this case it does so GET requests do not get forwarded to the primary and I end up seeing a stale value in the secondary datacenter.
It might be related to this Github issue Consul configs on secondary dc for federated datacenters - 1.9.3 · Issue #10356 · hashicorp/consul · GitHub.
I think this could happen if a config entry is written to the secondary data center before it joins the primary. Or maybe it happens if the primary goes down temporarily and config entries get written to the secondary datacenter before the primary recovers. Please let me know if this sounds possible.
This is hard to fix because PUT/DELETE requests DO get forwarded to the primary datacenter so it’s not possible to delete values from the secondary datacenter directly. I’ve had to take down the primary datacenter and then delete the values from the secondary in order to get the DELETE requests sent to the secondary datacenter.
Does any of this sound familiar? Am on the right track with this? I’m looking for a better way to handle this.
@blake any ideas? Does that sound familiar?