Correct way to connect to upstream that uses dynamic ports

Nomad
1

Hello, I want to run an upstream job that has dynamic ports and connect a downstream job to the upstream by using the NOMAD_UPSTREAM_ADDR_<destination_name> environment variables. But my job file does not work as expected.

Versions

$ nomad -v
Nomad v1.0.4 (9294f35f9aa8dbb4acb6e85fa88e3e2534a3e41a)
$ consul -v
Consul v1.9.1
Revision ca5c38943
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

Reproduction steps

  1. Start Nomad and Consul dev agents. (using two terminals, in reality)
$ sudo nomad agent -dev-connect
$ consul agent -dev
  1. Create test.nomad file like the below:
job "test" {
  datacenters = ["dc1"]

  group "caddy" {
    task "caddy" {
      driver = "docker"

      config {
        image = "caddy"
      }
    }

    network {
      mode = "bridge"

      # Below http port is not referenced now.
      port "http" {
        to = 80
      }
    }

    service {
      name = "caddy"
      port = 80 # I want to change this to "http" to use dynamic ports.

      connect {
        sidecar_service {}
      }
    }
  }

  group "downstream" {
    task "downstream" {
      driver = "docker"

      config {
        # It is just only curl that I need.
        image   = "node"
        command = "tail"
        args    = ["-f", "/dev/null"]
      }
    }

    network {
      mode = "bridge"
    }

    service {
      connect {
        sidecar_service {
          proxy {
            upstreams {
              destination_name = "caddy"
              local_bind_port  = 80
            }
          }
        }
      }
    }
  }
}

Note: Caddy is a web server written in Go and zero-conf Caddy docker container serves an instruction page at port 80 by default.

  1. Then executing like the below is successful:
$ nomad job run test.nomad
$ nomad alloc exec -task downstream -job test bash
# curl $NOMAD_UPSTREAM_ADDR_caddy -svo /dev/null
* Expire in 0 ms for 6 (transfer 0x559f8cbaafb0)
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x559f8cbaafb0)
* Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> Host: 127.0.0.1
> User-Agent: curl/7.64.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Content-Length: 12226
< Content-Type: text/html; charset=utf-8
< Etag: "qrkle19fm"
< Last-Modified: Wed, 14 Apr 2021 20:10:49 GMT
< Server: Caddy
< Date: Mon, 03 May 2021 09:16:35 GMT
< 
{ [12226 bytes data]
* Connection #0 to host 127.0.0.1 left intact
  1. Now I will make a small change by executing sed -i 's/ port = 80/ port = "http"/' test.nomad. The change it makes is:
@@ -20,7 +20,7 @@ job "test" {
 
     service {
       name = "caddy"
-      port = 80 # I want to set this to "http"
+      port = "http" # I want to set this to "http"
 
       connect {
         sidecar_service {}
  1. In this circumstance, the next curl execution would fail.
$ nomad job run test.nomad
$ nomad alloc exec -task downstream -job test bash
# curl $NOMAD_UPSTREAM_ADDR_caddy -svo /dev/null
* Expire in 0 ms for 6 (transfer 0x55e9c219efb0)
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x55e9c219efb0)
* Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> Host: 127.0.0.1
> User-Agent: curl/7.64.0
> Accept: */*
> 
* Recv failure: Connection reset by peer
* Closing connection 0

I think there is something I miss… but I don’t know what it is.

Thanks.

2

Hi @lens0021 :wave:

I think that what you are missing is setting the address_mode of your service to alloc. By default, Nomad will advertise the host port associated with the allocation (the random port), but for Connect, since everything lives in the same bridge network, you want to advertise the actual port your allocation is listening on.

I’m not sure if you already have it, but you will also need to assign the port to your Docker task.

So the changes that I think you need to make are these:

job "test" {
  datacenters = ["dc1"]

  group "caddy" {
    task "caddy" {
      driver = "docker"

      config {
        image = "caddy"
+       ports = ["http"]
      }
    }

    network {
      mode = "bridge"

      # Below http port is not referenced now.
      port "http" {
        to = 80
      }
    }

    service {
      name         = "caddy"
+     port         = "http"
+     address_mode = "alloc"

      connect {
        sidecar_service {}
      }
    }
  }

Give it a try and let me know how it goes :slightly_smiling_face:

1 Like
3

It works not only in my example job file but also in my actual working job file with Traefik as a load balancer! Thanks a lot, @lgfa29

1 Like