Port mapping with Nomad and Consul Connect

Nomad
,
1

Hi,

I am looking for help to get basic port mapping working. I am aiming to use Nomand with Consul Connect to manage traffic between a set of microservices.

The following job spec based on the countdash example works fine. counter-dashboard uses counter-api. counter-api is advertised in Consul as a service with a dynamic port (which is desired) and I pass that port into the container using ${NOMAD_PORT_port_api}. The web service binds to the dynamic port and it works.

However, I would prefer not to have to pass dynamic ports into every container I configure. I would prefer to leave Postgres on 5432, MySQL on 3306, web services on 80, etc. and port map the dynamic port to these default ports.

So the second job spec attempts to do that port mapping but it doesn’t work. According to the documentation here this seems to be the way to do it:

What am I doing wrong? Is this a bug, should this work?

I am using the following versions of Nomad/Consul/CNI:

Nomad v0.12.7 (6147cb578794cb2d0c35d68fe1791728a09bb081)

Consul v1.8.2
Revision ba7d9435e
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

CNI Plugins v0.8.6

Working example, dynamic port is passed into counter-api:

    job "countdash" {
       datacenters = ["dc1"]
       group "api" {
         network {
           mode = "bridge"
           port "port_api" { }
         }

         service {
           name = "count-api"
           port = "port_api"

           connect {
             sidecar_service {}
           }
         }

         task "web" {
           driver = "docker"
           config {
             image = "hashicorpnomad/counter-api:v1"
           }
           env {
             PORT = "${NOMAD_PORT_port_api}"
           }
         }
       }

       group "dashboard" {
         network {
           mode ="bridge"
           port "http" {
             static = 9002
             to     = 9002
           }
         }

         service {
           name = "count-dashboard"
           port = "9002"

           connect {
             sidecar_service {
               proxy {
                 upstreams {
                   destination_name = "count-api"
                   local_bind_port = 8085
                 }
               }
             }
           }
         }

         task "dashboard" {
           driver = "docker"
           env {
             COUNTING_SERVICE_URL = "http://${NOMAD_UPSTREAM_ADDR_count_api}"
           }
           config {
             image = "hashicorpnomad/counter-dashboard:v1"
           }
         }
       }
     }

This doesn’t work, note the to = 9001 on the network port and ports = ["port_api"] on the task config. The counter-api service will bind to 9001 by default. The documentation https://www.nomadproject.io/docs/drivers/docker#using-the-port-map suggests that this should work:

job "countdash" {
   datacenters = ["dc1"]
   group "api" {
     network {
       mode = "bridge"
       port "port_api" { to = 9001 }
     }

     service {
       name = "count-api"
       port = "port_api"

       connect {
         sidecar_service {}
       }
     }

     task "web" {
       driver = "docker"
       config {
         image = "hashicorpnomad/counter-api:v1"
         ports = ["port_api"]
       }
     }
   }

   group "dashboard" {
     network {
       mode ="bridge"
       port "http" {
         static = 9002
         to     = 9002
       }
     }

     service {
       name = "count-dashboard"
       port = "9002"

       connect {
         sidecar_service {
           proxy {
             upstreams {
               destination_name = "count-api"
               local_bind_port = 8085
             }
           }
         }
       }
     }

     task "dashboard" {
       driver = "docker"
       env {
         COUNTING_SERVICE_URL = "http://${NOMAD_UPSTREAM_ADDR_count_api}"
       }
       config {
         image = "hashicorpnomad/counter-dashboard:v1"
       }
     }
   }
 }

Any ideas why the second example does not work?

2

In case anyone finds this useful I found a solution to this (although, I have not tested this across our cluster yet, single node only) by following the advice here https://github.com/hashicorp/nomad/issues/7229#issuecomment-649418198.

The change to the second example above is to also set the value of local_service_port on the service side car proxy to the same value as the to port (9001 in this case).

job "countdash" {
   datacenters = ["dc1"]
   group "api" {
     network {
       mode = "bridge"
       port "port_api" { to = 9001 }
     }

     service {
       name = "count-api"
       port = "port_api"

       connect {
         sidecar_service {
           proxy {
             local_service_port = 9001
           }
         }
       }
     }

     task "web" {
       driver = "docker"
       config {
         image = "hashicorpnomad/counter-api:v1"
         ports = ["port_api"]
       }
     }
   }

   group "dashboard" {
     network {
       mode ="bridge"
       port "http" {
         static = 9002
         to     = 9002
       }
     }

     service {
       name = "count-dashboard"
       port = "9002"

       connect {
         sidecar_service {
           proxy {
             upstreams {
               destination_name = "count-api"
               local_bind_port = 8085
             }
           }
         }
       }
     }

     task "dashboard" {
       driver = "docker"
       env {
         COUNTING_SERVICE_URL = "http://${NOMAD_UPSTREAM_ADDR_count_api}"
       }
       config {
         image = "hashicorpnomad/counter-dashboard:v1"
       }
     }
   }
 }
2 Likes
3

I was banging my head against the wall on this. This works great! Thanks.

4

Thank you as well. I was about to open an official ticket with HashiSupport when I saw this discussion and was able to workaround the problem. I still believe this is a bug as the default should always take the to value instead of the public side of the port.

5

I found this out the hard way a while back too. It seems like an oversight that this is a required value. I’m not sure under what circumstances you’d want this to not bind to the same port that the service is using. Having to ensure these magic numbers match in two places you can’t use variables is bound to fail.

6

I think the tutorial was written to show how to migrate existing code to consul connect. It does not take into account that some code is created as consul connect from very beginning. So it keeps some backward compatibility.

Whenever one create a dynamic port nomad is creating iptable rule for it on host interface. It is useless for a service hidden behind the sidecar proxy. It may be dangerous if the port is actually open.

So my personal rules for services inside consul connect mesh:

  • no static ports
  • no job.group.network.port
  • always job.group.network.mode = “bridge”
  • no docker.config.ports
  • only service.port as a number (points directly to the service)
  • always bind the service to 127.0.0.1

Of course if the service needs to be reachable from outside the consul connect mesh then the rules needs to be soften (ingest / load balancing for example)