wan federated multi-cloud kubernetes clusters - Error from server: cross-datacenter failover is only supported in the default pa

Consul
1

Overview of the Issue

I am trying multi-cloud consul mesh wan federation with AWS EKS and Azure AKS, following the article here Deploy federated multi-cloud Kubernetes clusters | Terraform | HashiCorp Developer.

Everything seems fine, the wan members are connected and able to access the services across the cloud.

Node                   Address          Status  Type    Build   Protocol  DC     Partition  Segment
consul-server-0.aws    10.0.1.38:8302   alive   server  1.16.0  2         aws    default    <all>
consul-server-0.azure  10.244.1.4:8302  alive   server  1.16.0  2         azure  default    <all>

However, when I try to deploy a service resolver for DC failover, it gives me the following error.

---
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceResolver
metadata:
  name: facts-api-backend
spec:
  connectTimeout: 5s
  failover:
    '*':
      datacenters: ['azure']

$ kubectl apply -f serviceresolver.yaml --context eks
Error from server: error when creating "serviceresolver.yaml": admission webhook "mutate-serviceresolver.consul.hashicorp.com" denied the request: serviceresolver.consul.hashicorp.com "facts-api-backend" is invalid: spec.failover[*].datacenters: Invalid value: []string{"azure"}: cross-datacenter failover is only supported in the default partition

The services example I am using is from this github repo - GitHub - jacobmammoliti/consul-multicloud-demo

Reproduction Steps

Follow the article : Deploy federated multi-cloud Kubernetes clusters | Terraform | HashiCorp Developer

Deploy the example services : consul-multicloud-demo/kubernetes at main · jacobmammoliti/consul-multicloud-demo · GitHub

Apply the service-resolver.yaml - consul-multicloud-demo/kubernetes/serviceresolver.yaml at main · jacobmammoliti/consul-multicloud-demo · GitHub

Consul info for both Client and Server

Server info 
agent:
        check_monitors = 0
        check_ttls = 0
        checks = 0
        services = 0
build:
        prerelease = 
        revision = 192df66a
        version = 1.16.0
        version_metadata = 
consul:
        acl = disabled
        bootstrap = true
        known_datacenters = 2
        leader = true
        leader_addr = 10.0.1.38:8300
        server = true
raft:
        applied_index = 259
        commit_index = 259
        fsm_pending = 0
        last_contact = 0
        last_log_index = 259
        last_log_term = 2
        last_snapshot_index = 0
        last_snapshot_term = 0
        latest_configuration = [{Suffrage:Voter ID:6580154e-6c92-837b-adfe-f294261f2a57 Address:10.0.1.38:8300}]
        latest_configuration_index = 0
        num_peers = 0
        protocol_version = 3
        protocol_version_max = 3
        protocol_version_min = 0
        snapshot_version_max = 1
        snapshot_version_min = 0
        state = Leader
        term = 2
runtime:
        arch = amd64
        cpu_count = 2
        goroutines = 259
        max_procs = 2
        os = linux
        version = go1.20.4
serf_lan:
        coordinate_resets = 0
        encrypted = false
        event_queue = 1
        event_time = 2
        failed = 0
        health_score = 0
        intent_queue = 1
        left = 0
        member_time = 2
        members = 1
        query_queue = 0
        query_time = 1
serf_wan:
        coordinate_resets = 0
        encrypted = false
        event_queue = 0
        event_time = 1
        failed = 0
        health_score = 0
        intent_queue = 0
        left = 0
        member_time = 2
        members = 2
        query_queue = 0
        query_time = 1

I don’t understand this issue and couldn’t find anything related to this, please help to check if this is a bug or a config issue.