Cross-Namespace Consul Cluster Membership Issue on EKS

Consul
1

Hi,

We have deployed two separate Consul clusters (each with 3 replicas) in different namespaces within the same Amazon EKS cluster:

  • Namespace abc: Consul server version consul/consul:1.15.4
  • Namespace xyz: Consul server version bitnami/consul:10.20.0

Both Consul deployments were intended to operate independently, but we have observed that the Consul servers from one namespace are becoming members of the Consul servers in the other namespace. This behavior is unexpected as the clusters should be isolated from each other.

Configuration Details:

  • Kubernetes Deployment Type: StatefulSets with 3 replicas each
  • Helm Charts Used: Official HashiCorp chart in abc, Bitnami chart in xyz

Troubleshooting Steps Taken:

  1. Test Cluster: We attempted to reproduce this issue in a separate test environment, deploying Consul in two different namespaces with identical configurations. In the test cluster, the Consul servers remained isolated and operated as independent clusters as expected.

Could you provide any insights on potential causes for this cross-namespace membership behavior, or suggest additional steps to ensure full cluster isolation?