Consul Connect Failover to Service

Consul
,
1

I would like to use consul connect to direct Database TCP traffic to a set of proxysql instances, but if they all fail, the traffic should failover to going directly to the database which is external ( RDS ).

I have connect working and the application is able to hit localhost and it routes traffic to the proxysql as expected through the service ( upstream: database1 ).
I have the RDS Database registered in Consul as an external node ( database1-rds ) and it is also registered in the mesh as a terminating gateway. When I change the connect config to point directly at the terminating gateway ( upstream: database1-rds ), the application can communicate through the mesh through the terminating gateway to the RDS database as expected.

I have a “service-resolver” put in with a Failover section:

Kind          = "service-resolver"
Name          = "database1"
Failover = {
  "*" = {
    Service  = "database1-rds"
  }
}

Under the service “database1”, under Routing, the UI shows “Service Failover” as database1-rds under Resolvers.

I don’t have anything set for Router or Splitters.
No acl’s are running in this test environment.
Intentions are allow all in this test environment.
Everything is in the same datacenter.

Consul version: v1.11.4
Nomad version: v1.2.6
Envoy containers are the default version it downloads.

The problem is that when I simulate the proxysql failure ( proxysql stop ), the application can not talk to the database. I expected, with the failover, that the traffic would automagically start going to the database1-rds service.

I’m sure there is a config or something I’m missing so any help would be greatly appreciated.

Thanks!