Nomad 101 : difficulty understanding networking between two tasks in one group

Nomad
1

Hi,

I am testing Nomad and trying to figure out the network basics but I seem to miss something there.

I am running two docker containers : one with rabbitmq as an mqtt broker, and another simple web server to mimic authentication.

job "brokerauth" {
    datacenters = ["dc1"]

    type = "service"

    group "broker" {
        count = 1

        network {
            port "management" {
                static = 15672
                to = 15672
            }
            port "tcp" {
                static = 5672
                to = 5672
            }
            port "mqtt" {
                static = 1883
                to = 1883
            }
            port "web" {
                static = 8080
                to = 8080
            }
        }

         task "rabbitmq" {
            driver = "docker"

            config {
                image = "rabbitmq:3.11.8-management"
                volumes = [
                    "local/enabled_plugins:/etc/rabbitmq/enabled_plugins",
                    "local/rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf",
                    ]
                ports = ["management", "tcp", "mqtt"]
            }

            resources {
                cpu = 300
                memory = 300
            }

            template {
                change_mode = "restart"
                data = <<EOH
                [rabbitmq_management, rabbitmq_mqtt,rabbitmq_auth_backend_http].
                EOH
                
                destination = "local/enabled_plugins"
                }

            template {
                change_mode = "restart"
                data = <<EOH
                loopback_users.guest = false

                # standard tcp configuration
                listeners.tcp.default = 5672

                default_vhost = /
                default_user = guest
                default_pass = guest

                default_permissions.configure = .*
                default_permissions.read = .*
                default_permissions.write = .*

                ## Tags for default user
                default_user_tags.administrator = true

                ## Define other tags like this:
                default_user_tags.management = true
                default_user_tags.custom_tag = true

                # admin management ui tool 
                management.tcp.port = 15672
            
                # mqtt configuration
                mqtt.listeners.tcp.default = 1883
                mqtt.allow_anonymous  = false
                mqtt.vhost            = /
                mqtt.exchange         = amq.topic

                # authentication configuration
                auth_backends.1 = http
                auth_backends.2 = internal
                auth_http.http_method   = post
                auth_http.user_path     = http://{{ env "NOMAD_ADDR_web" }}/user
                auth_http.vhost_path    = http://{{ env "NOMAD_ADDR_web" }}/vhost
                auth_http.resource_path = http://{{ env "NOMAD_ADDR_web" }}/resource
                auth_http.topic_path    = http://{{ env "NOMAD_ADDR_web" }}/topic

                #debugging
                log.console = true
                log.console.level = debug
                EOH
                
                destination = "local/rabbitmq.conf"
                }
        }
        
        task "webserver" {
            driver = "docker"

            config {
                image = "nicolascrochet7/myserver:latest"
                ports = ["web"]
            }
        }
    }
}

I am starting with the basics by putting those two tasks in the same group. Allocation is healthy and each task performs as intended on their own.

When connecting, rabbitmq is supposed to post request my web server for authentication. But NOMAD_ADDR_web has a connection timeout and I am not seeing any request attempt on my web server.

When two tasks are on the same group, what IP and port address should I be using so one can access the other ?

2

You can try setting “network_mode = “host”” in the config-stanza of your rabbitmq-task, then it should work. (I could curl 127.0.0.1:8080 when setting the value, tested with windows and nomad in dev-mode)

This is a good post for explaining networking with nomad: Karan Sharma | Understanding Networking in Nomad

3

Thank you for your response,

My bad, I thought grouping task would put them by default in a shared network and that I should not use “network_mode” stanza in the docker config because they would be “unable to reach other containers in the task group”.

4

I would also like to understand the use case OP posted.

network_mode = “host” makes ports accessible publicly on the host, as described here: [question] Nomad simple networking between tasks · Issue #9777 · hashicorp/nomad · GitHub but that’s often not what is desired, and isn’t what I want.

e.g. a stack with a webserver, app server, and database. Only the webserver port should be accessible publicly; the app server and database should be isolated within the task group and not accessible publicly.

The webserver should be able to contact the app server (and potentially the db server) within its task group, and the app server can contact the db server. But nothing in another task group, or on any other host, should be able to contact the db server or app server.

What does that configuration look like? I have hunted around all over the place but everything seems to suggest either network_mode = “host” to get it to work or to use Consul Connect and have a full service mesh.

This “isolated networking within the task group stack” use case is fairly straightforward to achieve with docker-compose and it seems like it should be possible in Nomad but I’m stumped about how to achieve it without adding a bunch of extra config settings to use a custom Docker network (network_mode = “my_custom_docker_network”).

It feels like it should be possible with network_mode = “bridge” but I can’t find example configs of how to do it, and haven’t been able to work it out on my own. Anyone have any clues?