Issue in Nomad Multi-Cloud Federated Cluster

Hi Community,

I’ve created two nomad-consul colocated clusters in both AWS and GCP one in each. They are working good individually. I tried federating the clusters in AWS and GCP together. I’ve enabled both inbound and outbound TCP/UDP access in both the clusters. I’m facing the below challenge. .It’d be helpful if someone could assist on this.

Error determining leaders: 1 error(s) occurred:

*** Region “us-east-2”: Unexpected response code: 500 (rpc error: failed to get conn: dial tcp i/o timeout)**

No Path to region

Hi @Dhinesh_Prabakaran,

Could you provide a little more detail on how the clusters are configured, particularly around their IP addressing and the connectivity between AWS and GCP?

The error message contains i/o timeout which indicates you’re trying to connect the federated clusters using internal/private IP addresses which might not be routable between providers.

jrasell and the Nomad team

Hi @jrasell.
The VMs in the clusters ( AWS and GCP) are created using custom AMIs which contains both nomad and consul installed. The IP address of the server node in the GCP which is used to join this cluster with the cluster running in AWS is a public ip. But in the error statement we see that the private ip being dialled and eventually timed out.

Now, instead of creating two separate clusters and trying to federate them, I’ve created a cluster in AWS (3 servers and 2 clients) and a few clients nodes alone in GCP. Using the below command, I’ve configured the GCP client nodes to join with the server nodes in AWS.

nomad node config -update-servers IP:4647

I’m able to see the list of servers in AWS to which the client nodes in GCP are connected. But, upon trying to query the node status from GCP, I’m facing the below challenge.

Error querying node status: Unexpected response code: 500 (rpc error: No path to region)

Hi @Dhinesh_Prabakaran,

Could you share the region configuration parameter you have for your AWS and GCP instances as well as the full command you are attempting to run?

jrasell and the Nomad team

Hi @jrasell ,

Please find below the config files of both nomad and consul server and client node respectively. I’ve tried the below three commands:

1. nomad server join
2. nomad node config -update-servers :4647
3. consul join


AWS - Nomad_Server_Config File:
Screenshot from 2022-07-23 13-35-19

AWS - Nomad_Client_Config File:

Screenshot from 2022-07-23 13-37-36

GCP - Nomad_Server_Config File:
Screenshot from 2022-07-23 13-40-57

GCP - Nomad_Client_Config File:

Screenshot from 2022-07-23 13-38-30


AWS- Consul_Server_Config file:
Screenshot from 2022-07-23 13-47-20

AWS- Consul_Client_config file

Screenshot from 2022-07-23 13-39-45

GCP-Consul_Server_Config file:

GCP-Consul_Client_Config file:

Screenshot from 2022-07-23 13-39-45

There are two use cases:

  1. Multi-Cluster Federation - Federating the cluster in AWS and GCP. Upon trying, I’m facing the below challenge:

After a shortwhile, the below challenge:

  1. Multi_Region Federation - Tried federqating yhe cluster in AWS and a set of client nodes alone from GCP. Federated succesfully but after federation, when tried to view the nodes from GCP, I’m able to see only the Server nodes and not the client nodes.

Kindly assist!

Hey @jrasell . Could you please look into the details and provide a solution ?

Team. This question is pending since a week. Could you please look into it and provide a solution soon ? @jrasell

Hi @Dhinesh_Prabakaran,

When running nomad node status you are not supplying a region identifier, therefore the default value of global is used, a region which you do not have. If you wish to list the nodes within us-east-2 (AWS) I would suggest trying the nomad node status -region=us-east-2 command; the command nomad node status -region=us-east1 will list the nodes within the us-east1 (GCP) region.

jrasell and the Nomad team