RabbitMQ with Consul Connect using Nomad

Nomad
1

TLDR: how 2+ instances of the same task can communicate between each other while using connect?

I was wondering … is it possible to somehow deploy multiple instances of RabbitMQ talking to each other using consul connect?
I’ve found some examples without using connect with some advanced magic I’m not yet familiar with.

I’ve managed to run 2 separate instances using connect - that’s easy.
But I did not manage to connect them to each other so they would form a cluster.

So the questions I have:
Do you know and can tell how to do this thing? (or point me to some already existing resource)
Do you know something with a similar problem? (elasticsearch comes to mind)
What are some keywords I might be looking for? (apparently “consul connect” “nomad” “rabbitmq” isn’t it)

Edit:
From what I’ve read there’s no access to Consul from the container itself so there’s no point in trying to use rabbitmq_peer_discovery_consul. In bridge mode anyway.

I could use a template to use the cluster_formation.peer_discovery_backend = classic_config and generate the config, but I’m not sure if it can handle IPs instead of hostnames. I’d also have to have a separate upstreams section for each instance so the ports don’t clash. And those different ports might also be an issue for the rabbit.

What I have now:

job "rabbitmq" {
    datacenters = ["dc1"]
    type = "service"
    group "rabbitmq" {
        count = 2
        network {
            mode = "bridge"
        }
        service {
            name = "rabbitmq"
            port = "5672"
            connect {
                sidecar_service {
                    proxy {
# this is currently useless
                        upstreams {
                            destination_name = "consul"
                            local_bind_port = "5200"
                        }
                    }
                }
            }
        }
        service {
            name = "rabbitmq-management"
            tags = ["management"]
            port = "15672"
            connect {
                sidecar_service {}
            }
        }
        task "rabbitmq" {
            resources {
                cpu = 1000
                memory = 1000
            }
            template {
                data =<<EOH
                [rabbitmq_management,rabbitmq_peer_discovery_consul].
                EOH
                destination = "local/enabled_plugins"
            }
            template {
                data =<<EOH
                cluster_formation.peer_discovery_backend = consul
# next 2 lines doesn't work since it can't resolve it
                cluster_formation.consul.host = {{ env "NOMAD_UPSTREAM_IP_consul" }}
                cluster_formation.consul.port = {{ env "NOMAD_UPSTREAM_PORT_consul" }}
                cluster_formation.consul.svc = rabbitmq-cluster
                cluster_formation.consul.acl_token = xxx # I'm testing now, it's fine :)
                EOH
                destination = "local/rabbitmq.conf"
            }
            driver = "docker"
            config {
                image = "rabbitmq:3.8-management-alpine"
                volumes = ["local/enabled_plugins:/etc/rabbitmq/enabled_plugins", "local/rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf"]
            }
        }
    }
    group "rabbitmq-management-ingress" {
        network {
            mode = "bridge"
            port "inbound" {
                static = 15672
                to = 15672
            }
        }
        service {
            name = "rabbitmq-management-ingress"
            port = "15672"
            connect {
                gateway {
                    proxy {}
                    ingress {
                        listener {
                            port = 15672
                            protocol = "tcp"
                            service {
                                name = "rabbitmq-management"
                            }
                        }
                    }
                }
            }
        }
    }
}
2

Hi @DejfCold :wave:

I don’t have a lot of experience with RabbitMQ, so I apologies if I say anything wrong.

From their docs, it seems like each node must be able to resolve the other’s hostname. My gut feeling is that this goes against the way Consul Connect operates because it will hide each of the nodes behind a single proxy.

Here’s a simple job without Consul Connect:

job "rabbitmq" {
  datacenters = ["dc1"]
  type        = "service"

  group "rabbitmq" {
    count = 2

    network {
      mode = "host"

      port "rabbitmq" {
        to = 5672
      }

      port "rabbitmq-management" {
        to = 15672
      }
    }

    task "rabbitmq" {
      driver = "docker"

      config {
        image = "rabbitmq:3.8-management-alpine"
        volumes = [
          "local/enabled_plugins:/etc/rabbitmq/enabled_plugins",
          "local/rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf",
        ]
      }

      resources {
        cpu    = 300
        memory = 300
      }

      template {
        data = <<EOH
[rabbitmq_management,rabbitmq_peer_discovery_consul].
EOH

        destination = "local/enabled_plugins"
      }

      template {
        data        = <<EOH
cluster_formation.peer_discovery_backend = consul
cluster_formation.consul.host = {{ env "attr.unique.network.ip-address" }}
EOH
        destination = "local/rabbitmq.conf"
      }
    }
  }
}

It uses the host network to be able to reach the local Consul agent.

Is Consul Connect something that you require in your deployment?

2 Likes
3

Hi @lgfa29!
Thanks for spending some time on this.
I was experimenting with Nomad a bit this week and when I’ve read your answer, I got an idea.

I used to think that I’d need connect on everything, but that’s apparently not true. I’d prefer it now, but it’s not entirely required.

I’ve found there’s a thing called terminating proxy.

Do you think a terminating proxy could be usable for this?

I came up with this, but have no way to test it at this moment:

job "rabbitmq" {
    datacenters = ["dc1"]
    type = "service"
    group "rabbitmq" {
        count = 2
        network {
            mode = "host"
            port "rabbitmq" {
                to = 5672
            }
            port "rabbitmq-management" {
                to = 15672
            }
        }
        # No service declarations here as
        # rabbitmq registers itself with consul as
        # "rabbitmq" and "rabbitmq_management" services
        task "rabbitmq" {
            resources {
                cpu = 300
                memory = 300
            }
            template {
                data =<<EOH
                [rabbitmq_management,rabbitmq_peer_discovery_consul].
                EOH
                destination = "local/enabled_plugins"
            }
            template {
                data = <<EOH
                cluster_formation.peer_discovery_backend = consul
                cluster_formation.consul.host = {{ env "attr.unique.network.ip-address" }}
                EOH
                destination = "local/rabbitmq.conf"
            }
            driver = "docker"
            config {
                image = "rabbitmq:3.8-management-alpine"
                volumes = [
                    "local/enabled_plugins:/etc/rabbitmq/enabled_plugins",
                    "local/rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf"
                ]
            }
            volume_mount {
                volume = "data"
                destination = "/var/lib/rabbitmq"
            }
        }
        volume "data" {
            type = "host"
            source = "rabbitmq"
            read_only = false
        }
    }
    group "rabbitmq-terminating" {
        network {
            mode = "bridge"
        }
        service {
            connect {
                gateway {
                    proxy {}
                    terminating {
                        service {
                            name = "rabbitmq"
                        }
                    }
                }
            }
        }
    }
}

I assume, I could then use service.connect.sidecar_service.proxy.upstreams.destination_name = "rabbitmq" in some other Connect enabled task and nomad would route it through the terminating proxy properly, right?

4

I know im dragging up an older thread here, but this is exactly the problem im having right now as well (just not with RabbitMQ). I’m looking to set up Keycloak inside Consul Connect, behind a Traefik ingress gateway for TLS termination. To get a cluster working the instances need to be able to talk to each other, but there doesnt seem to be a way to set an upstream to other instances of the same task (not to mention the possibility of a dynamic number of instances as well). Can this be done without a native consul connect application?

1 Like