i have multi DC
one of DC1 service “echo” calls DC2 service
and this mesh-gateway mode is local
{
"name": "echo",
"port": 9198,
"checks": [
{
"http": "http://127.0.0.1:9198",
"Interval": "10s"
}
],
"connect": {
"sidecar_service": {
"port": 21002,
"proxy": {
"upstreams": [
{
"destination_name": "echo-dc2-hc",
"datacenter": "dc2",
"local_bind_port": 11006,
"config": {
"passive_health_check": {
"interval": "3s",
"max_failures": "1"
}
}
]
}
}
}
}
in DC2, echo-dc2-hc service has two instance
when one of echo-dc2-hc instance is health down , in DC2 mesh-gateway log ,
i can see this logs
[2024-02-13 06:17:48.418][7][debug][config] [source/extensions/config_subscription/grpc/new_grpc_mux_impl.cc:92] Received DeltaDiscoveryResponse for type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment at version
[2024-02-13 06:17:48.418][7][debug][upstream] [source/common/upstream/upstream_impl.cc:453] transport socket match, socket default selected for host with address .....
[2024-02-13 06:17:48.418][7][debug][upstream] [source/common/upstream/upstream_impl.cc:453] transport socket match, socket default selected for host with address ......
[2024-02-13 06:17:48.418][7][debug][upstream] [source/extensions/clusters/eds/eds.cc:368] EDS hosts or locality weights changed for cluster: echo-dc2-hc.default.dc2.internal.aaaaaa.consul current hosts 2 priority 0
[2024-02-13 06:17:48.418][59][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1221] membership update for TLS cluster echo-dc2-hc.default.dc2.internal.aaaaaa.consul added 0 removed 0
[2024-02-13 06:17:48.418][60][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1221] membership update for TLS cluster echo-dc2-hc.default.dc2.internal.aaaaaa.consul added 0 removed 0
[2024-02-13 06:17:48.418][62][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1221] membership update for TLS cluster echo-dc2-hc.default.dc2.internal.aaaaaa.consul added 0 removed 0
[2024-02-13 06:17:48.419][70][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1221] membership update for TLS cluster echo-dc2-hc.default.dc2.internal.aaaaaa.consul added 0 removed 0
[2024-02-13 06:17:48.419][7][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1221] membership update for TLS cluster echo-dc2-hc.default.dc2.internal.aaaaaa.consul added 0 removed 0
[2024-02-13 06:17:48.419][64][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1221] membership update for TLS cluster echo-dc2-hc.default.dc2.internal.aaaaaa.consul added 0 removed 0
[2024-02-13 06:17:48.419][72][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1221] membership update for TLS cluster echo-dc2-hc.default.dc2.internal.aaaaaa.consul added 0 removed 0
[2024-02-13 06:17:48.419][66][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1221] membership update for TLS cluster echo-dc2-hc.default.dc2.internal.aaaaaa.consul added 0 removed 0
[2024-02-13 06:17:48.419][7][debug][config] [source/extensions/config_subscription/grpc/new_delta_subscription_state.cc:262] Delta config for type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment accepted with 1 resources added, 0 removed
[2024-02-13 06:17:48.419][68][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1221] membership update for TLS cluster echo-dc2-hc.default.dc2.internal.aaaaaa.consul added 0 removed 0
first time, this log interval was 30s
so, i searched consul document, i found “performance. grpc_keepalive_timeout” configuration
when i changed to 10s, DC2 mesh-gateway logs , looks like configuration is working
but, in DC1,
when i check “echo” envoy proxy logs
it still interval is 30s
log look like this
[2024-02-13 06:17:05.706][7][debug][config] [source/common/config/new_grpc_mux_impl.cc:90] Received DeltaDiscoveryResponse for type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment at version
[2024-02-13 06:17:05.707][7][debug][upstream] [source/common/upstream/upstream_impl.cc:256] transport socket match, socket default selected for host with address ….
[2024-02-13 06:17:05.707][7][debug][upstream] [source/common/upstream/eds.cc:360] EDS hosts or locality weights changed for cluster: echo-dc2-hc.default.dc2.internal.aaaaa.consul current hosts 1 priority 0
[2024-02-13 06:17:05.707][7][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1116] membership update for TLS cluster echo-dc2-hc.default.dc2.internal.aaaaa.consul added 0 removed 0
[2024-02-13 06:17:05.707][38][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1116] membership update for TLS cluster echo-dc2-hc.default.dc2.internal.aaaaa.consul added 0 removed 0
[2024-02-13 06:17:05.707][41][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1116] membership update for TLS cluster echo-dc2-hc.default.dc2.internal.aaaaa.consul added 0 removed 0
[2024-02-13 06:17:05.707][49][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1116] membership update for TLS cluster echo-dc2-hc.default.dc2.internal.aaaaa.consul added 0 removed 0
[2024-02-13 06:17:05.707][46][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1116] membership update for TLS cluster echo-dc2-hc.default.dc2.internal.aaaaa.consul added 0 removed 0
[2024-02-13 06:17:05.707][43][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1116] membership update for TLS cluster echo-dc2-hc.default.dc2.internal.aaaaa.consul added 0 removed 0
[2024-02-13 06:17:05.707][47][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1116] membership update for TLS cluster echo-dc2-hc.default.dc2.internal.aaaaa.consul added 0 removed 0
[2024-02-13 06:17:05.707][7][debug][config] [source/common/config/new_delta_subscription_state.cc:255] Delta config for type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment accepted with 1 resources added, 0 removed
[2024-02-13 06:17:05.707][39][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1116] membership update for TLS cluster echo-dc2-hc.default.dc2.internal.aaaaa.consul added 0 removed 0
[2024-02-13 06:17:05.707][51][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1116] membership update for TLS cluster echo-dc2-hc.default.dc2.internal.aaaaa.consul added 0 removed 0
i tried many things , in that way
i found “passive_health_check” configuration
this configuration, it removed unhealthy service
but it didn’t help me perfectly
"passive_health_check": {
"interval": "3s",
"max_failures": "1"
}
do you know the way, in multi DC,
when remote DC service is partially unhealthy , changing caller envoy refresh-interval-time configuration
i am using hashicorp/consul:1.17.2 version