I have setup a consul service mesh using wan federated Mesh gateway following this guide Federation Between VMs and Kubernetes | Consul by HashiCorp.
My primary cluster is a VM cluster and secondary cluster is on k8s. I am able to see all the members of my secondary cluster in WAN pool but for some reason only few members of my primary cluster gets listed on the WAN pool.
This issue is very random. If i restart my primary cluster, the nodes which were part of the wan pool goes missing and the nodes which were not present before seems to be visible.
I could not get any relevant errors also relating to it in consul server logs.
The nodes which do not appear to be in WAN pool and if i do
consul catalog datacenters from those nodes, it shows only local datacenter, which proves that it is not part of the WAN pool.
What is wrong here ?