How to know/setup port range exposed by sidecar proxies

Nomad
1

Hi.
We are trying to use sidecar_service to connect 2 services together with nomad.

I used the sample job from nomad init -connect command, and ran it into a 2 client node cluster (dashboard and api tasks are each deployed to their own host)
It works perfectly as long as we allow any TCP traffic between the 2 nodes.
But I we want to allow only a whitelist of necessary ports, communication is obviously cut off.

OK, so I checked from nomad and consul documentation which ports to open, and allowed 21000-21255 port range according to Consul ports requirements.
But… It stil does not works, and I notice that the envoy container does not listen to a port within the previously documented range (27849 in my case)
screen

If I open this port btw my 2 host servers, it works again, like expected.
So my questions are:

  • which range of port is used by nomad while creating sidecar instances ? Is it documented elsewhere ?
  • Can I force a fixed port value for the sidecar task on my own ?

Thank you for any help.

2

Hi @ballinette!

The port range Nomad uses for dynamic ports is from 20000-32000.

There is not currently a way to configure this range.

You can use a static port for the sidecar task - but of course that comes with the overhead of managing the uniqueness of host port allocations yourself. To do so simply use a static port block, and map that into the network namespace. E.g.,

port "sidecar" {
  static = 20000
  to     = 20000
}
3

thanks @shoenig for your answer.

I was looking for a way to distinguish the dynamic ports on sidecar proxies, from dynamic ports on common services, in order to allow only the first ones in my firewall, but apparently, it is not possible, so for now I will allow the whole 20000-32000 range between nodes on my nomad cluster.

To clarify my first question: I have following job for example:

  group "database" {
    network {
       mode = "bridge"
       port "database" {
         to = 3306
       }
    }

    service {
      name = "database"
      port = 3306
      connect {
        sidecar_service {
        }
      }
    }

    task "database" {
// (....)
    }
}

Both “database” and “database-sidecar-proxy” services will have an exposed port within 20000-32000 range, and I cannot force the sidecar port to be inside a custom range, and the other port to be outside this custom range.

If I use static ports as you suggest as a workaround, it seems to work for the service itself, but not for the sidecar proxy…

4

Hi again.

I’ve just realised my misunderstanding: it is not necessary to expose the “database” port if I want to force all incoming requests to pass through the sidecar.

This is sufficient:

 group "database" {
    network {
       mode = "bridge"
    }

    service {
      name = "database"
      port = 3306
      connect {
        sidecar_service {
        }
      }
    }

    task "database" {
// (....)
    }
}

Then only the sidecar task is exposed to a dynamic port, which suits my needs.

1 Like