There are two clusters .
There are two different services/applications.
All public requests first go to a primary app in one cluster.
If the primary app knows the request (non 404 http response code) it responds appropriately.
If the primary app doesn’t know the request (404 http response code) it redirects the request the fallback secondary app running on another cluster.
Question is, is Consul the way to go here and can this scenario be implemented with Consul?
I’m assuming you’re using 404 as a generic – no answer. DNS doesn’t use HTTP codes.
My guess is that it can’t be with “just” consul, but I think you can setup your DNS to follow that logic and query consul A and then B to see if A can resolve if not then check B. The follow up issue would be how to set up the forwarders so that A or B don’t know then what to do.
This scenario cannot be implemented with Consul alone.
That is because Consul does not implement the concept of “if one server fails to handle the client’s request, send another copy of the client’s request to a different server”.
You need another technology added to the mix for that - some flavour of smart HTTP proxy/loadbalancer.