Service-resolver failover to secondary and tertiary datacenters

Consul Failover Service-Resolver

Consul Version: v1.11.3

Currently I have 3 datacenters all governed by Consul. All consul servers are healthy and aware of each other.

Goal: When I take one of my service pods offline and curl it I would like the ability for Consul to route the call to one of the other datacenters in my Consul configuration.

Experiment: When I delete the pod causing it to obviously not be able to serve the request when curl is executed I receive “no healthy upstream”

Following the documentation service-resolver

My service-resolver was applied to the primary and stated synced to the other datacenters.
Here is the service resolver.

kubectl get -n prd service-ci-prd -o yaml
kind: ServiceResolver
  annotations: |
      {"apiVersion":"","kind":"ServiceResolver","metadata":{"annotations":{},"name":"service-ci-prd","namespace":"prd"},"spec":{"connectTimeout":"15s","defaultSubset":"default","failover":{"*":{"datacenters":["k-dc-g1","k-dc-g3","k-dco-g1"]}},"subsets":{"default":{"filter":"Service.Meta.environment == prd"}}}}
  creationTimestamp: "2022-04-06T18:41:41Z"
  generation: 17
  name: service-ci-prd
  namespace: prd
  resourceVersion: "137557685"
  selfLink: /apis/
  uid: 9b5521a8-eb25-46e2-90c3-81434a5fc15f
  connectTimeout: 15s
  defaultSubset: default
      - k-dc-g1
      - k-dc-g3
      - k-dco-g1
      filter: Service.Meta.environment == prd
  - lastTransitionTime: "2022-04-08T18:23:06Z"
    status: "True"
    type: Synced
  lastSyncedTime: "2022-04-08T18:23:06Z"

In the consul UI in all datacenters it shows the dcs and the failover applied. However when the pod is killed and curled I receive no healthy upstream. Note it also Synced correctly per above. In the UI I am going to the service and then instances and under Meta I am using the data there environment prd. Does anyone else know what else I need to check in order to get the routing to work when the one instance in the primary is down to route to one of the others. For all intensive purposes it looks like it is aware of the other dcs and instances of the services.

Hi, I think this might be a bug. Would you be okay to create a github issue?