Hi,
Following up on my Nomad discovery, I am trying to connect two tasks in different groups.
Here is the job spec for this :
job "brokerauth" {
datacenters = ["dc1"]
type = "service"
meta {
run_uuid = "${uuidv4()}"
}
group "broker" {
count = 1
network {
port "management" {
to = 15672
}
port "tcp" {
to = 5672
}
port "mqtt" {
to = 1883
}
}
service {
name = "broker"
provider = "nomad"
port = "mqtt"
}
task "rabbitmq" {
driver = "docker"
config {
image = "rabbitmq:3.11.8-management"
volumes = [
"local/enabled_plugins:/etc/rabbitmq/enabled_plugins",
"local/rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf",
]
ports = ["management", "tcp", "mqtt"]
}
resources {
cpu = 300
memory = 300
}
template {
change_mode = "restart"
data = <<EOH
[rabbitmq_management, rabbitmq_mqtt,rabbitmq_auth_backend_http].
EOH
destination = "local/enabled_plugins"
}
template {
change_mode = "restart"
data = <<EOH
loopback_users.guest = false
# standard tcp configuration
listeners.tcp.default = 5672
default_vhost = /
default_user = guest
default_pass = guest
default_permissions.configure = .*
default_permissions.read = .*
default_permissions.write = .*
## Tags for default user
default_user_tags.administrator = true
## Define other tags like this:
default_user_tags.management = true
default_user_tags.custom_tag = true
# admin management ui tool
management.tcp.port = 15672
management.cors.allow_origins.1 = *
# mqtt configuration
mqtt.listeners.tcp.default = 1883
mqtt.allow_anonymous = false
mqtt.vhost = /
mqtt.exchange = amq.topic
mqtt.durable_queue_type = quorum
# authentication configuration
auth_backends.1 = http
auth_backends.2 = internal
auth_http.http_method = post
{{ range nomadService "auth" }}
auth_http.user_path = http://{{ .Address }}:{{ .Port }}/user
auth_http.vhost_path = http://{{ .Address }}:{{ .Port }}/vhost
auth_http.resource_path = http://{{ .Address }}:{{ .Port }}/resource
auth_http.topic_path = http://{{ .Address }}:{{ .Port }}/topic
{{ end }}
#debugging
log.console = true
log.console.level = debug
EOH
destination = "local/rabbitmq.conf"
}
}
}
group "auth" {
count = 1
network {
port "web" {
to = 8080
}
}
service {
name = "auth"
port = "web"
provider = "nomad"
}
task "webserver" {
driver = "docker"
config {
image = "nicolascrochet7/myserver:latest"
ports = ["web"]
}
}
}
}
When reaching out through mqtt to the rabbitmq task, the rabbitmq task should http post to the second “auth” task. Unfortunately, while ranging through nomadService works well in the template (I could curl the url), the auth service sees no request.
I’ve tried bridge, host, but nothing works.
Any idea of what might go wrong ? and why ?
Hi @nicolascrochet7 can you provide a way of testing whether the configuration is working or not? I converted your job to use bridge networking mode in the job file below, but I dunno how to tell if it works or not.
job "brokerauth" {
datacenters = ["dc1"]
type = "service"
meta {
run_uuid = "${uuidv4()}"
}
group "broker" {
count = 1
network {
mode = "bridge"
port "management" {
to = 15672
}
port "tcp" {
to = 5672
}
port "mqtt" {
to = 1883
}
}
service {
name = "broker"
provider = "nomad"
port = "mqtt"
}
task "rabbitmq" {
driver = "docker"
config {
image = "rabbitmq:3.11.8-management"
volumes = [
"local/enabled_plugins:/etc/rabbitmq/enabled_plugins",
"local/rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf",
]
}
resources {
cpu = 300
memory = 300
}
template {
change_mode = "restart"
data = <<EOH
[rabbitmq_management, rabbitmq_mqtt,rabbitmq_auth_backend_http].
EOH
destination = "local/enabled_plugins"
}
template {
change_mode = "restart"
data = <<EOH
loopback_users.guest = false
# standard tcp configuration
listeners.tcp.default = 5672
default_vhost = /
default_user = guest
default_pass = guest
default_permissions.configure = .*
default_permissions.read = .*
default_permissions.write = .*
## Tags for default user
default_user_tags.administrator = true
## Define other tags like this:
default_user_tags.management = true
default_user_tags.custom_tag = true
# admin management ui tool
management.tcp.port = 15672
management.cors.allow_origins.1 = *
# mqtt configuration
mqtt.listeners.tcp.default = 1883
mqtt.allow_anonymous = false
mqtt.vhost = /
mqtt.exchange = amq.topic
mqtt.durable_queue_type = quorum
# authentication configuration
auth_backends.1 = http
auth_backends.2 = internal
auth_http.http_method = post
{{ range nomadService "auth" }}
auth_http.user_path = http://{{ .Address }}:{{ .Port }}/user
auth_http.vhost_path = http://{{ .Address }}:{{ .Port }}/vhost
auth_http.resource_path = http://{{ .Address }}:{{ .Port }}/resource
auth_http.topic_path = http://{{ .Address }}:{{ .Port }}/topic
{{ end }}
#debugging
log.console = true
log.console.level = debug
EOH
destination = "local/rabbitmq.conf"
}
}
}
group "auth" {
count = 1
network {
mode = "bridge"
port "web" {
to = 8080
}
}
service {
name = "auth"
port = "web"
provider = "nomad"
}
task "webserver" {
driver = "docker"
config {
image = "nicolascrochet7/myserver:latest"
}
}
}
}
Hi Seth,
Of course,
I am using an mqtt client to hit connection on rabbitmq on the “mqtt” port. Otherwise, a simple hello world amqp connection could also do it on the “tcp” port. I am just more familiar with mqtt.
For mqtt, you can use the mosquitto local client. Installation on linux :
sudo apt search mosquitto
sudo apt install mosquitto mosquitto-clients
Then this command would initiate a connect and publish command :
mosquitto_pub -h 127.0.0.1 -p [rabbitmq host dynamic port] -u namka -P namka -q 1 -t topic -m "hello"
The (namka, namka) are the (user, password) combination that the auth web server will respond positively to.
You should see the rabbitmq logs attempting reaching out to the auth web server, but on my case with a connection timeout and no post request in the auth web server logs.
The two tasks work well when in the same group.
Btw, I could make this work by putting each task in a bridge network with a specific hostname.
Here is the job spec :
job "brokerauth" {
datacenters = ["dc1"]
type = "service"
meta {
run_uuid = "${uuidv4()}"
}
group "broker" {
count = 1
network {
mode = "bridge"
hostname = "pont"
port "management" {
to = 15672
}
port "tcp" {
static = 5672
to = 5672
}
port "mqtt" {
static = 1883
to = 1883
}
}
service {
name = "broker"
provider = "nomad"
port = "mqtt"
}
task "rabbitmq" {
driver = "docker"
config {
image = "rabbitmq:3.11.8-management"
volumes = [
"local/enabled_plugins:/etc/rabbitmq/enabled_plugins",
"local/rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf",
]
ports = ["management", "tcp", "mqtt"]
}
resources {
cpu = 300
memory = 300
}
template {
change_mode = "restart"
data = <<EOH
[rabbitmq_management, rabbitmq_mqtt,rabbitmq_auth_backend_http].
EOH
destination = "local/enabled_plugins"
}
template {
change_mode = "restart"
data = <<EOH
loopback_users.guest = false
# standard tcp configuration
listeners.tcp.default = 5672
default_vhost = /
default_user = guest
default_pass = guest
default_permissions.configure = .*
default_permissions.read = .*
default_permissions.write = .*
## Tags for default user
default_user_tags.administrator = true
## Define other tags like this:
default_user_tags.management = true
default_user_tags.custom_tag = true
# admin management ui tool
management.tcp.port = 15672
management.cors.allow_origins.1 = *
# mqtt configuration
mqtt.listeners.tcp.default = 1883
mqtt.allow_anonymous = false
mqtt.vhost = /
mqtt.exchange = amq.topic
mqtt.durable_queue_type = quorum
# authentication configuration
auth_backends.1 = http
auth_backends.2 = internal
auth_http.http_method = post
{{ range nomadService "auth" }}
auth_http.user_path = http://{{ .Address }}:{{ .Port }}/user
auth_http.vhost_path = http://{{ .Address }}:{{ .Port }}/vhost
auth_http.resource_path = http://{{ .Address }}:{{ .Port }}/resource
auth_http.topic_path = http://{{ .Address }}:{{ .Port }}/topic
{{ end }}
#debugging
log.console = true
log.console.level = debug
EOH
destination = "local/rabbitmq.conf"
}
}
}
group "auth" {
count = 1
network {
mode = "bridge"
hostname = "pont"
port "web" {
to = 8080
}
}
service {
name = "auth"
port = "web"
provider = "nomad"
}
task "webserver" {
driver = "docker"
config {
image = "nicolascrochet7/myserver:latest"
ports = ["web"]
}
}
}
}
I would still be interesting in understanding why things work in one case and not in the other, since I’m mostly trying to figure out Nomad.