Problems with gRPC load balancing

Consul
1

We have nomad cluster (3 consul machines and 4-5 nomad runners)
We have http service that calls gRPC service.
We are using service mesh with consul and envoy (default)

Sadly load balancing for gRPC is not working, from logs i see that only 1 gRPC service is getting all the requests, other services only get health check calls and nothing else.

HTTP service is calling gRPC via service mesh

   ....
     connect {
        sidecar_service {
          proxy {
            upstreams {
              local_bind_port  = 8070
              destination_name = "grpcservice-grpcservice"
            }
            ...
          }
        }
      }

      template {
        destination = "services.env"
        env         = true
        change_mode = "restart"
        data        = <<EOH
GRPC_SERVICE_ADDRESS={{(printf "%s" (env "NOMAD_UPSTREAM_ADDR_grpcservice"))}}
EOH
      }
...

gRPC service

...
  service {
      name = "grpcservice"
      port = "grpc"
      tags = ["grpc"]

      connect {
        sidecar_service {
          proxy {
            upstreams {
              local_bind_port  = 8070
              destination_name = "some_other_service"
            }
          }
        }
      }
....

Does any1 has idea where the issue could be or how to troubleshoot it.

2

Hi @VladimirZD,

With gRPC services to properly load balance the requests, you should be setting the gRPC services protocol type to grpc using a service-defaults config entry.

ref: Service defaults configuration entry reference | Consul | HashiCorp Developer

In your case, you will have to apply a config entry as shown below:

Kind      = "service-defaults"
Name      = "grpcservice"
Protocol  = "grpc"

Once you apply this, if still the requests are not spread, make sure you restart the downstream application.

Please let me know if this works for you.

1 Like
3

Hey, thank you very much. I did basic test and looks like it is working.
Tomorrow i will run load test will last longer so i will confirm this, but at first look looks like it is working.

I was checking documentation, and it looks i will need to do this for each grpc service i have, because if i am doing some kind of global change then it will apply to all proxies and i have mixed services (http, grpc).

When checking docs for sidecar settings i noticed this config setting it looks like it could be used to set this on service level for proxy

Thnx

1 Like