Nomad - communication between docker containers within one task seems not to be working

I am using Apache Kafka V3.1 in docker and trying to orchestrate it with Nomad. I am facing a problem creating a distributed cluster.

the goal is to have 3 broker/controller nodes on 3 EC2 Instances

:~$ nslookup broker.service.brain.consul

Name:   broker.service.brain.consul
Name:   broker.service.brain.consul
Name:   broker.service.brain.consul

from inside one of the Nomad Client Instances:

IPv4 address for docker0:
IPv4 address for ens5:
IPv4 address for nomad:

here is the relevant Nomad Job configuration

job "kafka" {
  datacenters = ["stream"]
  type = "service"
  group "broker" {
    count = 3
    service {
      name = "broker"
      port = "9092"
      tags = ["kafka","broker"]
      connect {
        sidecar_service {}
    network {
      mode = "bridge"
      hostname = "${attr.unique.hostname}"
      dns {
        servers = [""]
      port "broker" {
        static = 9092
        to     = 9092
      port "controler" {
        static = 9093
        to     = 9093
    task "broker" {

      driver = "docker"
      config {

        image = ""
        volumes = ["files/"]

        ports = [

the after rendering from template looks as following: (the changes accross 3 brokers)


However, The cluster is not able to start and it seems to be a connections issue.

[2022-01-24 01:31:15,405] ERROR [BrokerLifecycleManager id=2] Shutting down because we were unable to register with the controller quorum. (kafka.server.BrokerLifecycleManager)
[2022-01-24 01:31:15,407] INFO [BrokerLifecycleManager id=2] registrationTimeout: shutting down event queue. (org.apache.kafka.queue.KafkaEventQueue)
[2022-01-24 01:31:15,407] INFO [BrokerLifecycleManager id=2] Transitioning from STARTING to SHUTTING_DOWN. (kafka.server.BrokerLifecycleManager)
[2022-01-24 01:31:15,408] INFO [BrokerServer id=2] Transition from STARTING to STARTED (kafka.server.BrokerServer)
[2022-01-24 01:31:15,408] INFO [BrokerToControllerChannelManager broker=2 name=heartbeat]: Shutting down (kafka.server.BrokerToControllerRequestThread)
[2022-01-24 01:31:15,409] INFO [BrokerToControllerChannelManager broker=2 name=heartbeat]: Stopped (kafka.server.BrokerToControllerRequestThread)
[2022-01-24 01:31:15,410] INFO [BrokerToControllerChannelManager broker=2 name=heartbeat]: Shutdown completed (kafka.server.BrokerToControllerRequestThread)
[2022-01-24 01:31:15,412] ERROR [BrokerServer id=2] Fatal error during broker startup. Prepare to shutdown (kafka.server.BrokerServer)
	at java.base/java.util.concurrent.CompletableFuture.cancel(
	at kafka.server.BrokerLifecycleManager$
	at org.apache.kafka.queue.KafkaEventQueue$
	at java.base/
[2022-01-24 01:31:15,417] INFO [BrokerServer id=2] Transition from STARTED to SHUTTING_DOWN (kafka.server.BrokerServer)




[2022-01-24 02:02:19,304] INFO [RaftManager nodeId=2] Disconnecting from node 1 due to socket connection setup timeout. The timeout value is 10341 ms. (org.apache.kafka.clients.NetworkClient)
[2022-01-24 02:02:19,306] INFO [RaftManager nodeId=2] Disconnecting from node 3 due to socket connection setup timeout. The timeout value is 11036 ms. (org.apache.kafka.clients.NetworkClient)
[2022-01-24 02:02:20,100] INFO [RaftManager nodeId=2] Re-elect as candidate after election backoff has completed (org.apache.kafka.raft.KafkaRaftClient)

I did try to set the listeners to match the new docker hostname hostname = "${attr.unique.hostname}", or to the EC2 host IP but those didn’t help neither.

I’ve spent few days on this puzzle but currently I am out of ideas. Would appreciate any help on this issue.