Can nomad service discovery use dockers internal IPs to avoid exposing the container ports?

Nomad
1

Hi,
due to dockers way of doing a lot with iptables when forwarding ports I’ve been thinking, whether it is possible to skip the port forwarding with service discovery. So far my research says no, but I am not sure enough about that.

I can communicate with docker containers on their internal exposed but not forwarded ports from within the same network without the need to forward them with -p PORT:PORT.

So if a db container with an IP 172.10.200.3 listens on port 3309, each container in the same network can communicate with 172.10.200.3:3309 without requiring use a mapped port on the host (= no need to forward 3309).

I know how I can create a job, where the docker port is not forwarded, but is it possible to create a nomad service, that does not point to the nomad host IP, but rather to the internal docker container IP?

When I create this job

job "port-expose-tests" {
  datacenters = ["dc1"]
  type        = "service"

  group "port-expose-tests"  {
    count = 1

    task "port-expose-tests"{
        driver = "docker"

        config {
            image = "ealen/echo-server"
            # This will expose it to host, but commenting it out, breaks nomad service discovery
            #ports = ["echo"]
        }

        service {
            name = "port-server"
            port = "echo"
            provider = "nomad"
        }
    }
    network {
      port "echo" {
        static = 80
      }
    }
  }
}

I setup the port, on which the internal container listens to, but this of course breaks the service discovery. The service will be registered at IP_of_nomad:80 instead of IP_of_<port-expose-container>:80

This also breaks with nomads “load balancer”, but why this design, though?

If I could get the internal IP of port-expose-tests via consul template from any other job, I would eliminate the need of forwarding the container port to the host.

So can somebody confirm, that in theory I could use internal IPs of the containers but, there is no proper way to actually advertise the container’s internal IP in a service?

2

I do not think there is a way to do that. You could just register the service in consul separately.

I do not see much use to it anyway. As the docker network IP addresses are local within the host, other hosts will not able to communicate with the service. It is a setup viable only for a single-node cluster. Why would you do that?

If you want secure service-to-service communication, on Nomad side this is solved with consul connect and sidecar proxies.

3

Not sure why you are trying to do it this way, but regardless you’d set the address_mode = driver