Nomad 101 : difficulty understanding networking between two tasks in one group

Nomad
1

Hi,

I am testing Nomad and trying to figure out the network basics but I seem to miss something there.

I am running two docker containers : one with rabbitmq as an mqtt broker, and another simple web server to mimic authentication.

job "brokerauth" {
    datacenters = ["dc1"]

    type = "service"

    group "broker" {
        count = 1

        network {
            port "management" {
                static = 15672
                to = 15672
            }
            port "tcp" {
                static = 5672
                to = 5672
            }
            port "mqtt" {
                static = 1883
                to = 1883
            }
            port "web" {
                static = 8080
                to = 8080
            }
        }

         task "rabbitmq" {
            driver = "docker"

            config {
                image = "rabbitmq:3.11.8-management"
                volumes = [
                    "local/enabled_plugins:/etc/rabbitmq/enabled_plugins",
                    "local/rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf",
                    ]
                ports = ["management", "tcp", "mqtt"]
            }

            resources {
                cpu = 300
                memory = 300
            }

            template {
                change_mode = "restart"
                data = <<EOH
                [rabbitmq_management, rabbitmq_mqtt,rabbitmq_auth_backend_http].
                EOH
                
                destination = "local/enabled_plugins"
                }

            template {
                change_mode = "restart"
                data = <<EOH
                loopback_users.guest = false

                # standard tcp configuration
                listeners.tcp.default = 5672

                default_vhost = /
                default_user = guest
                default_pass = guest

                default_permissions.configure = .*
                default_permissions.read = .*
                default_permissions.write = .*

                ## Tags for default user
                default_user_tags.administrator = true

                ## Define other tags like this:
                default_user_tags.management = true
                default_user_tags.custom_tag = true

                # admin management ui tool 
                management.tcp.port = 15672
            
                # mqtt configuration
                mqtt.listeners.tcp.default = 1883
                mqtt.allow_anonymous  = false
                mqtt.vhost            = /
                mqtt.exchange         = amq.topic

                # authentication configuration
                auth_backends.1 = http
                auth_backends.2 = internal
                auth_http.http_method   = post
                auth_http.user_path     = http://{{ env "NOMAD_ADDR_web" }}/user
                auth_http.vhost_path    = http://{{ env "NOMAD_ADDR_web" }}/vhost
                auth_http.resource_path = http://{{ env "NOMAD_ADDR_web" }}/resource
                auth_http.topic_path    = http://{{ env "NOMAD_ADDR_web" }}/topic

                #debugging
                log.console = true
                log.console.level = debug
                EOH
                
                destination = "local/rabbitmq.conf"
                }
        }
        
        task "webserver" {
            driver = "docker"

            config {
                image = "nicolascrochet7/myserver:latest"
                ports = ["web"]
            }
        }
    }
}

I am starting with the basics by putting those two tasks in the same group. Allocation is healthy and each task performs as intended on their own.

When connecting, rabbitmq is supposed to post request my web server for authentication. But NOMAD_ADDR_web has a connection timeout and I am not seeing any request attempt on my web server.

When two tasks are on the same group, what IP and port address should I be using so one can access the other ?

2

You can try setting “network_mode = “host”” in the config-stanza of your rabbitmq-task, then it should work. (I could curl 127.0.0.1:8080 when setting the value, tested with windows and nomad in dev-mode)

This is a good post for explaining networking with nomad: Karan Sharma | Understanding Networking in Nomad

3

Thank you for your response,

My bad, I thought grouping task would put them by default in a shared network and that I should not use “network_mode” stanza in the docker config because they would be “unable to reach other containers in the task group”.