How to make jobs communicate over loopback interface

Nomad
1

I have 2 containers in a nomad group. How can I make it so they communicate over a loopback interface without registering the service in consul?

job "foobar" {
    datacenters = ["dc1"]
    group "foo" {

        volume "puppet-certs" {
            type = "host"
            config {
                source = "puppet-certs"
            }
        }
        task "corvus" {
            driver = "docker"
            config {
                image = "gitlab.example.com:4567/example/corvus:0.2.7"
                entrypoint = [" /go/bin/corvus"]
                args = [
                "--nodes",
                "redis.service.consul:6379",
                "--bind",
                "8379",
                "/dev/null"
                ]
            }
            resources {
                network {
                    port "corvus" {
                        static = "8379"
                    }
                }
            }
        }
        task "redis-cli" {
            driver = "docker"
            config {
                image = "redis:4.0.10"
                command = "/bin/sleep"
                args = ["10000"]
            }
        }

    }
}

I then log into the redis-cli and try and query the other container on the loopback

nomad alloc exec -task redis-cli -job foobar /bin/bash
$ redis-cli ping -p 8379  #no response
2

In Nomad 0.10 and newer, you can use consul connect to create envoy proxies that will seamlessly route traffice without having to create consul services

  1. Ensure that you have the CNI plugins already installed
  1. change the network mode to “bridge”.
    group "foo" {

        network {
            mode = "bridge"
        }
  1. Add an envoy proxy sidecar on the service you want to connect to (corvus)
            service {
                connect {
                    sidecar_service {}
                }
            }
  1. Register the destination port on the service you are connecting from (redis-cli)
            service {
                connect {
                    sidecar_service {
                        proxy {
                            upstreams {
                                destination_name = "corvus"
                                local_bind_port = 8379
                            }
                        }
                    }

                }
            }

A full example:

job "foobar" {
    datacenters = ["dc1"]
    group "foo" {

        network {
            mode = "bridge"
        }
        task "corvus" {
            driver = "docker"
            config {
                image = "gitlab.example.com:4567/example/ci-images/corvus:0.2.7"
                entrypoint = ["/go/bin/corvus"]
                args = [
                "--nodes",
                "corvus.service.consul:6379",
                "--bind",
                "8379",
                "/dev/null"
                ]
            }
            service {
                connect {
                    sidecar_service {}
                }
            }
        }
        task "redis-cli" {
            driver = "docker"
            config {
                image = "redis:4.0.10"
                command = "/bin/sleep"
                args = ["10000"]
            }
            service {
                connect {
                    sidecar_service {
                        proxy {
                            upstreams {
                                destination_name = "corvus"
                                local_bind_port = 8379
                            }
                        }
                    }

                }
            }
        }

    }
}

I can then connect into the ‘redis-cli’ task and run these commands

nomad alloc exec -task redis-cli -job foobar /bin/bash
apt update
apt install telnet net-tools
redis-cli -h localhost -p 8379 ping
PONG

Netstat also shows just the 1 available port as listening

netstat -plnt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:8379            0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:8379            0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:8379            0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:8379            0.0.0.0:*               LISTEN      -
1 Like