does not see DC services although wan members have Alive. if you look from a remote DC services
consul catalog nodes -datacenter=lg-stage-google-k8s-us-central1
Error listing nodes: Unexpected response code: 500 (Remote DC has no server currently reachable)
but at the same time:
consul members -wan | grep lg-stage-google-k8s-us-central1
consul-server-0.lg-stage-google-k8s-us-central1 10.20.2.56:8302 alive server 1.17.1 2 lg-stage-google-k8s-us-central1 default <all>
consul-server-1.lg-stage-google-k8s-us-central1 10.20.1.27:8302 alive server 1.17.1 2 lg-stage-google-k8s-us-central1 default <all>
consul-server-2.lg-stage-google-k8s-us-central1 10.20.0.26:8302 alive server 1.17.1 2 lg-stage-google-k8s-us-central1 default <all>
if you look at the services locally:
consul catalog nodes -datacenter=lg-stage-google-k8s-us-central1
Node ID Address DC
consul-server-0 07571214 10.20.2.56 lg-stage-google-k8s-us-central1
consul-server-1 b380837f 10.20.1.27 lg-stage-google-k8s-us-central1
consul-server-2 983a222c 10.20.0.26 lg-stage-google-k8s-us-central1
gke-lg-stage-gke-lg-stage-gke-5fd36748-1w6h-virtual 10.10.0.7 lg-stage-google-k8s-us-central1
gke-lg-stage-gke-lg-stage-gke-7883d6cf-6903-virtual 10.10.0.8 lg-stage-google-k8s-us-central1
gke-lg-stage-gke-lg-stage-gke-98acdb60-cpzw-virtual 10.10.0.6 lg-stage-google-k8s-us-central1
then they exist. I installed it with these settings
global:
name: consul
datacenter: lg-stage-google-k8s-us-central1
tls:
enabled: true
caCert:
secretName: consul-federation
secretKey: caCert
caKey:
secretName: consul-federation
secretKey: caKey
acls:
manageSystemACLs: true
replicationToken:
secretName: consul-federation
secretKey: replicationToken
federation:
enabled: true
k8sAuthMethodHost: https://3.3.3.3
primaryDatacenter: bhs5
primaryGateways:
- 2.2.2.2:8443
- 2.2.2.2:8443
gossipEncryption:
secretName: consul-federation
secretKey: gossipEncryptionKey
connectInject:
enabled: true
replicas: 3
disruptionBudget:
enabled: true
maxUnavailable: 1
meshGateway:
enabled: true
replicas: 3
logLevel: "debug"
service:
enabled: true
type: LoadBalancer
port: 8443
annotations: |
cloud.google.com/l4-rbs: "enabled"
networking.gke.io/internal-load-balancer-allow-global-access: "true"
additionalSpec: |
loadBalancerIP: 1.1.1.1
externalTrafficPolicy: Local
server:
replicas: 3
extraConfig: |
{
"primary_datacenter": "bhs5",
"connect": {
"enable_mesh_gateway_wan_federation": true
},
"log_level": "DEBUG"
}
gke settings:
Location type Regional
Shielded GKE nodes Enabled
Network 10.10.0.0/24
Cluster Pod IPv4 range pod-range (10.20.0.0/16)
IPv4 service range 10.30.0.0/20
setup gke:
resource "google_container_cluster" "primary" {
name = "${var.project_id}-gke"
location = var.region
# We can't create a cluster with no node pool defined, but we want to only use
# separately managed node pools. So we create the smallest possible default
# node pool and immediately delete it.
remove_default_node_pool = true
initial_node_count = 1
#node_locations = var.zones
min_master_version = var.fixed_master_version
release_channel {
channel = "UNSPECIFIED"
}
maintenance_policy {
recurring_window {
start_time = "2025-06-22T01:00:00Z"
end_time = "2025-06-22T05:00:00Z"
recurrence = "FREQ=DAILY"
}
}
deletion_protection = false
network = google_compute_network.vpc.name
subnetwork = google_compute_subnetwork.subnet.name
private_cluster_config {
enable_private_nodes = true
enable_private_endpoint = false
master_global_access_config {
enabled = true
}
}
ip_allocation_policy {
cluster_secondary_range_name = "pod-range"
services_secondary_range_name = "service-range"
}
addons_config {
http_load_balancing {
disabled = false
}
gcp_filestore_csi_driver_config {
enabled = false
}
}
}
resource "google_container_node_pool" "primary_nodes" {
name = google_container_cluster.primary.name
location = var.region
cluster = google_container_cluster.primary.name
version = var.fixed_master_version
node_count = var.gke_num_nodes
autoscaling {
min_node_count = var.gke_num_nodes
max_node_count = var.gke_max_num_nodes
location_policy = "BALANCED"
}
management {
auto_repair = true
auto_upgrade = false
}
node_config {
oauth_scopes = [
"https://www.googleapis.com/auth/logging.write",
"https://www.googleapis.com/auth/monitoring",
]
labels = {
env = var.project_id
nodepool = google_container_cluster.primary.name
}
machine_type = var.gke_type_nodes
tags = ["gke-node", "${var.project_id}-gke"]
metadata = {
disable-legacy-endpoints = "true"
}
}
}
firewall settings:
resource "google_compute_firewall" "allow-pod-traffic" {
name = "${var.project_id}-allow-pod-traffic"
network = google_compute_network.vpc.name
allow {
protocol = "all"
}
source_ranges = [
var.kubernetes_subnet,
var.kubernetes_secondary_ip_range_pod,
var.kubernetes_secondary_ip_range_service
]
direction = "INGRESS"
priority = 1000
}
resource "google_compute_firewall" "allow-pod-in-traffic" {
name = "${var.project_id}-allow-pod-in-traffic"
network = google_compute_network.vpc.name
allow {
protocol = "all"
}
destination_ranges = [
var.kubernetes_subnet,
var.kubernetes_secondary_ip_range_pod,
var.kubernetes_secondary_ip_range_service
]
direction = "EGRESS"
priority = 1000
}
resource "google_compute_firewall" "allow_glb_health_checks" {
name = "allow-glb-health-checks"
network = google_compute_network.vpc.name
project = var.project_id
allow {
protocol = "tcp"
ports = ["8300", "8301", "8302", "8443", "8500", "8501", "8502", "8503"]
}
allow {
protocol = "udp"
ports = ["8301", "8302", "8600"]
}
# Google Cloud health checkers and GLB source IPs
source_ranges = ["0.0.0.0/0"]
}
maybe someone has encountered this? how to raise secondary dc?