Health check failing on gRPC when using Connect

Nomad
1

We are trying to set up Nomad/Envoy service mesh.
Everything is installed and configured as in documentation and example from docs is working.

When we deploy gRPC service, health checks are failing, only solution we have found is to specify ports in network stanza.
This is not working

job "grpcTest" {
  datacenters = ["dc1"]
  type        = "service"

  update {
    max_parallel      = 1
    auto_revert       = true
    auto_promote      = true
    canary            = 1
    healthy_deadline  = "30s"
    progress_deadline = "2m"
  }

  group "grpc" {
    count = 1

    network {
      mode = "bridge"
    }

    service {
      name = "grpc-service"
      port = "50051"
      tags = ["grpc"]

      connect {
        sidecar_service {}
      }

      check {
        name     = "grpc-service_dynamicport"
        expose   = true
        type     = "grpc"
        interval = "10s"
        timeout  = "2s"
      }
    }

    task "grpc" {
      driver = "docker"
      config {
        image = ".."
        args = [
          "grpc",
        ]
        
      }

      env {
      }
    }
  }
}

This is working

job "grpcTest" {
  datacenters = ["dc1"]
  type        = "service"

  update {
    max_parallel      = 1
    auto_revert       = true
    auto_promote      = true
    canary            = 1
    healthy_deadline  = "30s"
    progress_deadline = "2m"
  }

  group "grpc" {
    count = 2

    network {
      mode = "bridge"
      port "grpc" {
      }
    }

    service {
      name = "grpc-service"
      port = "grpc"
      tags = ["grpc"]

      connect {
        sidecar_service {}
      }

      check {
        name     = "grpc-service_dynamicport"
        expose   = true
        type     = "grpc"
        interval = "10s"
        timeout  = "2s"
      }
    }

    task "grpc" {
      driver = "docker"
      config {
        image = ".."
        args = [
          "grpc",
        ]
      }
      env {
      }
    }
  }
}

I was expecting that I don’t need to define port in network as I have it in service section, so i am wondering if there is some kind of problem here which is solved by opening port in network part.

2

Hi @VladimirZD. Thanks for using Nomad.

This is by design! Here’s are some resources for you to help you understand Nomad networking.

I hope that helps!

3

Hi @DerekStrickland I will go over documentation, but I have quick question.
Are you saying this is right way to do it?

Thnx

4

I am saying it is “a” right way to do it :smiling_face: There is more than one way you could configure your network, but yes, this approach is completely valid.

1 Like