We are looking to migrate our in-house built orchestration and deployment system that runs a few hundred (also in-house written) applications to Nomad+Consul.
Our infra has multiple datacenters (physical DC, no cloud) with fairly large latency between them (120+ ms rtt as per Consul stats), however from the application logic perspective 2 or more datacenters are a “single unit”.
Given the network constraints of Consul and Nomad (primarily Consul), we see the topology we can use for the rollout as: 1 Consul+Nomad cluster per datacenter, with both Consul and Nomad federated across groups of datacenters.
Now that is all nice and works great for basic things, however, if we want to have application integrate with Consul natively (which for 90% of our applications is a simple spring-boot-consul library drop-in), then we are stuck with applications talking to only their local Consul cluster. That can work for KV data, but Service Catalog (registration, discovery, health checks etc) are useless then. Otherwise applications will need to be rewritten to be Consul aware (so not just library drop-in anymore) and to query all known Consul datacenters and then query each one for the information they need.
Am I missing something about setup? Is there any better way to set it up which won’t require rewriting several hundred apps (that can take years).