Port mapping with Nomad and Consul Connect

Hi,

I am looking for help to get basic port mapping working. I am aiming to use Nomand with Consul Connect to manage traffic between a set of microservices.

The following job spec based on the countdash example works fine. counter-dashboard uses counter-api. counter-api is advertised in Consul as a service with a dynamic port (which is desired) and I pass that port into the container using ${NOMAD_PORT_port_api}. The web service binds to the dynamic port and it works.

However, I would prefer not to have to pass dynamic ports into every container I configure. I would prefer to leave Postgres on 5432, MySQL on 3306, web services on 80, etc. and port map the dynamic port to these default ports.

So the second job spec attempts to do that port mapping but it doesn’t work. According to the documentation here this seems to be the way to do it:

What am I doing wrong? Is this a bug, should this work?

I am using the following versions of Nomad/Consul/CNI:

Nomad v0.12.7 (6147cb578794cb2d0c35d68fe1791728a09bb081)

Consul v1.8.2
Revision ba7d9435e
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

CNI Plugins v0.8.6

Working example, dynamic port is passed into counter-api:

    job "countdash" {
       datacenters = ["dc1"]
       group "api" {
         network {
           mode = "bridge"
           port "port_api" { }
         }

         service {
           name = "count-api"
           port = "port_api"

           connect {
             sidecar_service {}
           }
         }

         task "web" {
           driver = "docker"
           config {
             image = "hashicorpnomad/counter-api:v1"
           }
           env {
             PORT = "${NOMAD_PORT_port_api}"
           }
         }
       }

       group "dashboard" {
         network {
           mode ="bridge"
           port "http" {
             static = 9002
             to     = 9002
           }
         }

         service {
           name = "count-dashboard"
           port = "9002"

           connect {
             sidecar_service {
               proxy {
                 upstreams {
                   destination_name = "count-api"
                   local_bind_port = 8085
                 }
               }
             }
           }
         }

         task "dashboard" {
           driver = "docker"
           env {
             COUNTING_SERVICE_URL = "http://${NOMAD_UPSTREAM_ADDR_count_api}"
           }
           config {
             image = "hashicorpnomad/counter-dashboard:v1"
           }
         }
       }
     }

This doesn’t work, note the to = 9001 on the network port and ports = ["port_api"] on the task config. The counter-api service will bind to 9001 by default. The documentation https://www.nomadproject.io/docs/drivers/docker#using-the-port-map suggests that this should work:

job "countdash" {
   datacenters = ["dc1"]
   group "api" {
     network {
       mode = "bridge"
       port "port_api" { to = 9001 }
     }

     service {
       name = "count-api"
       port = "port_api"

       connect {
         sidecar_service {}
       }
     }

     task "web" {
       driver = "docker"
       config {
         image = "hashicorpnomad/counter-api:v1"
         ports = ["port_api"]
       }
     }
   }

   group "dashboard" {
     network {
       mode ="bridge"
       port "http" {
         static = 9002
         to     = 9002
       }
     }

     service {
       name = "count-dashboard"
       port = "9002"

       connect {
         sidecar_service {
           proxy {
             upstreams {
               destination_name = "count-api"
               local_bind_port = 8085
             }
           }
         }
       }
     }

     task "dashboard" {
       driver = "docker"
       env {
         COUNTING_SERVICE_URL = "http://${NOMAD_UPSTREAM_ADDR_count_api}"
       }
       config {
         image = "hashicorpnomad/counter-dashboard:v1"
       }
     }
   }
 }

Any ideas why the second example does not work?

In case anyone finds this useful I found a solution to this (although, I have not tested this across our cluster yet, single node only) by following the advice here https://github.com/hashicorp/nomad/issues/7229#issuecomment-649418198.

The change to the second example above is to also set the value of local_service_port on the service side car proxy to the same value as the to port (9001 in this case).

job "countdash" {
   datacenters = ["dc1"]
   group "api" {
     network {
       mode = "bridge"
       port "port_api" { to = 9001 }
     }

     service {
       name = "count-api"
       port = "port_api"

       connect {
         sidecar_service {
           proxy {
             local_service_port = 9001
           }
         }
       }
     }

     task "web" {
       driver = "docker"
       config {
         image = "hashicorpnomad/counter-api:v1"
         ports = ["port_api"]
       }
     }
   }

   group "dashboard" {
     network {
       mode ="bridge"
       port "http" {
         static = 9002
         to     = 9002
       }
     }

     service {
       name = "count-dashboard"
       port = "9002"

       connect {
         sidecar_service {
           proxy {
             upstreams {
               destination_name = "count-api"
               local_bind_port = 8085
             }
           }
         }
       }
     }

     task "dashboard" {
       driver = "docker"
       env {
         COUNTING_SERVICE_URL = "http://${NOMAD_UPSTREAM_ADDR_count_api}"
       }
       config {
         image = "hashicorpnomad/counter-dashboard:v1"
       }
     }
   }
 }