We have 6 consul clusters running version 1.2.4 .
We upgraded the first cluster (servers first, then clients) to version 1.6.9 and once the upgrade was completed we noticed that resolving of service tags from clients in the 1.6.9 cluster cross DC to the 1.2.4 version cluster return an empty list.
@nadler, for non-LTS branches, Consul only guarantees backward compatibility with the prior release. (See Consul Protocol Compatibility Promise | Consul | HashiCorp Developer).
We promise that every subsequent release of Consul will remain backwards compatible with at least one prior version.
Consul 1.2.x to 1.6.x is a pretty big jump. It’s possible that there were changes to the RPC communication that are affecting tag resolution. You should consider upgrading the servers in the other datacenters to 1.3.x or newer to see if that resolves the issue.
I recommend testing that first in a non-production environment before upgrading the production clusters.