We have nomad cluster (3 consul machines and 4-5 nomad runners)
We have http service that calls gRPC service.
We are using service mesh with consul and envoy (default)
Sadly load balancing for gRPC is not working, from logs i see that only 1 gRPC service is getting all the requests, other services only get health check calls and nothing else.
HTTP service is calling gRPC via service mesh
....
connect {
sidecar_service {
proxy {
upstreams {
local_bind_port = 8070
destination_name = "grpcservice-grpcservice"
}
...
}
}
}
template {
destination = "services.env"
env = true
change_mode = "restart"
data = <<EOH
GRPC_SERVICE_ADDRESS={{(printf "%s" (env "NOMAD_UPSTREAM_ADDR_grpcservice"))}}
EOH
}
...
gRPC service
...
service {
name = "grpcservice"
port = "grpc"
tags = ["grpc"]
connect {
sidecar_service {
proxy {
upstreams {
local_bind_port = 8070
destination_name = "some_other_service"
}
}
}
}
....
Does any1 has idea where the issue could be or how to troubleshoot it.