Terminating gateway returned 503 no_heathy_upstream for external service

Hello, I try to do simple nomad job, Go GET request from external services.

As what expected, I get status code response 200.

output from 'main.go'

``
2022/09/30 19:35:02 …
2022/09/30 19:35:02 200

When I uncomment resp, err := http.Get("http://127.0.0.1:9001") to be use inside nomad job. The nomad logs give me this,

2022/09/30 11:33:16 no healthy upstream
2022/09/30 11:33:16 503

What I do first, is I compile my go program and build docker image for test.nomad

env GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o ./main ./main.go

docker build . -t test:0.0.0

Next, I run consul and nomad in dev mode in separting terminal.

consul agent -dev -log-level DEBUG

sudo nomad agent -dev-connect -bind 0.0.0.0 -log-level DEBUG -config nomad-config/

Then, I register external services into consul. I follow this tutorial.

curl --request PUT --data @external.json localhost:8500/v1/catalog/register

consul config write external-defaults.hcl

After that I start terminating gateway and start the application

terminating.nomad
# terminating.nomad file

job "terminating" {
  datacenters = ["dc1"]
  type        = "service"

  group "gateway" {
    network {
      mode = "bridge"
    }

    service {
      name = "api-gateway"

      connect {
        gateway {
          proxy {}

          terminating {
            service {
              name = "learn"
            }
          }
        }
      }
    }
  }
}
test.nomad
# test.nomad file

job "test" {
  datacenters = ["dc1"]
  type        = "batch"

  group "app" {
    network {
      mode = "bridge"
    }

    service {
      name = "test"

      connect {
        sidecar_service {
          proxy {
            upstreams {
              destination_name   = "learn"
              local_bind_port    = 9001
              local_bind_address = "127.0.0.1"
            }
          }
        }
      }
    }

    task "app" {
      driver = "docker"

      config {
        image = "test:0.0.0"
      }

      resources {
        cpu    = 500
        memory = 256
      }
    }
  }
}

nomad run terminating.nomad

nomad run test.nomad

You can see here the at the topology from test services connecting to learn which is the external services through terminating gateway inside consul.

Here’s the log coming from terminating.nomad-log job.

terminating.nomad-log.txt (41.6 KB)

Thank you.

Ref: My tree for this project looks like this and can check my repo here.

├── Dockerfile
├── external-defaults.hcl
├── external.json
├── main
├── main.go
├── nomad-config
│   └── debug.hcl
├── terminating.nomad
└── test.nomad