I am using Apache Kafka V3.1 in docker and trying to orchestrate it with Nomad. I am facing a problem creating a distributed cluster.
the goal is to have 3 broker/controller nodes on 3 EC2 Instances
:~$ nslookup broker.service.brain.consul
Server: 127.0.0.1
Address: 127.0.0.1#53
Name: broker.service.brain.consul
Address: 30.10.12.52
Name: broker.service.brain.consul
Address: 30.10.11.8
Name: broker.service.brain.consul
Address: 30.10.13.172
from inside one of the Nomad Client Instances:
IPv4 address for docker0: 172.17.0.1
IPv4 address for ens5: 30.10.13.172
IPv4 address for nomad: 172.26.64.1
here is the relevant Nomad Job configuration
job "kafka" {
datacenters = ["stream"]
type = "service"
group "broker" {
count = 3
service {
name = "broker"
port = "9092"
tags = ["kafka","broker"]
connect {
sidecar_service {}
}
}
network {
mode = "bridge"
hostname = "${attr.unique.hostname}"
dns {
servers = ["172.17.0.1"]
}
port "broker" {
static = 9092
to = 9092
}
port "controler" {
static = 9093
to = 9093
}
}
...
task "broker" {
driver = "docker"
config {
image = "registry.gitlab.com/.../kafka"
volumes = ["files/server.properties:/kafka/config/kraft/server.properties"]
ports = [
"broker",
"controler"
]
...
the server.properties after rendering from template looks as following: (the node.id changes accross 3 brokers)
process.roles=broker,controller
node.id=2
controller.quorum.voters=1@30.10.11.8:9093,2@30.10.12.52:9093,3@30.10.13.172:9093
listeners=PLAINTEXT://:9092,CONTROLLER://:9093
advertised.listeners=PLAINTEXT://:9092
inter.broker.listener.name=PLAINTEXT
controller.listener.names=CONTROLLER
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
num.network.threads=3
num.io.threads=8
request.timeout.ms=60000
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/logs/kraft-combined-logs
However, The cluster is not able to start and it seems to be a connections issue.
[2022-01-24 01:31:15,405] ERROR [BrokerLifecycleManager id=2] Shutting down because we were unable to register with the controller quorum. (kafka.server.BrokerLifecycleManager)
[2022-01-24 01:31:15,407] INFO [BrokerLifecycleManager id=2] registrationTimeout: shutting down event queue. (org.apache.kafka.queue.KafkaEventQueue)
[2022-01-24 01:31:15,407] INFO [BrokerLifecycleManager id=2] Transitioning from STARTING to SHUTTING_DOWN. (kafka.server.BrokerLifecycleManager)
[2022-01-24 01:31:15,408] INFO [BrokerServer id=2] Transition from STARTING to STARTED (kafka.server.BrokerServer)
[2022-01-24 01:31:15,408] INFO [BrokerToControllerChannelManager broker=2 name=heartbeat]: Shutting down (kafka.server.BrokerToControllerRequestThread)
[2022-01-24 01:31:15,409] INFO [BrokerToControllerChannelManager broker=2 name=heartbeat]: Stopped (kafka.server.BrokerToControllerRequestThread)
[2022-01-24 01:31:15,410] INFO [BrokerToControllerChannelManager broker=2 name=heartbeat]: Shutdown completed (kafka.server.BrokerToControllerRequestThread)
[2022-01-24 01:31:15,412] ERROR [BrokerServer id=2] Fatal error during broker startup. Prepare to shutdown (kafka.server.BrokerServer)
java.util.concurrent.CancellationException
at java.base/java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2396)
at kafka.server.BrokerLifecycleManager$ShutdownEvent.run(BrokerLifecycleManager.scala:478)
at org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:174)
at java.base/java.lang.Thread.run(Thread.java:829)
[2022-01-24 01:31:15,417] INFO [BrokerServer id=2] Transition from STARTED to SHUTTING_DOWN (kafka.server.BrokerServer)
...
also
...
[2022-01-24 02:02:19,304] INFO [RaftManager nodeId=2] Disconnecting from node 1 due to socket connection setup timeout. The timeout value is 10341 ms. (org.apache.kafka.clients.NetworkClient)
[2022-01-24 02:02:19,306] INFO [RaftManager nodeId=2] Disconnecting from node 3 due to socket connection setup timeout. The timeout value is 11036 ms. (org.apache.kafka.clients.NetworkClient)
[2022-01-24 02:02:20,100] INFO [RaftManager nodeId=2] Re-elect as candidate after election backoff has completed (org.apache.kafka.raft.KafkaRaftClient)
I did try to set the listeners to match the new docker hostname hostname = "${attr.unique.hostname}", or to the EC2 host IP but those didn’t help neither.
I’ve spent few days on this puzzle but currently I am out of ideas. Would appreciate any help on this issue.