I’m trying to use Jaeger for distributed tracing.
I have one Jaeger job running and exposing port 6831 which it uses to listen for UDP traffic of events related to tracing.
I have another job running a Python service that sends events to an upstream over a sidecar proxy with a local_bind_port.
When I have these two tasks in separate job files, the Jaeger agent never receives the trace events.
When I combine the two tasks into the same job file, the Jaeger agent receives the trace events.
My assumption was that Consul Connect abstracted away networking so that applications didn’t need to do anything different to talk to another service. I thought I could just set the sidecar service with an upstream proxy and a local bind port and the rest would be taken care of.
Can anyone clarify why one of these cases works and the other doesn’t?
Here is the Jaeger job file.
job "jaeger-tracing" {
datacenters = ["dc1"]
type = "system"
group "jaeger" {
network {
mode = "bridge"
port "compact_thrift" {
to = 6831
static = 6831
}
port "frontend" {
to = 16686
}
}
service {
name = "tracing"
port = "compact_thrift"
connect {
sidecar_service {}
}
}
task "jaeger" {
driver = "docker"
env {
SPAN_STORAGE_TYPE = "badger"
}
config {
image = "jaegertracing/all-in-one:1.21.0"
ports = ["compact_thrift", "frontend", "collector"]
}
}
}
}
And here is the service job file (Jaeger sections commented out. Things work when I uncomment those lines and run Jaeger in the same group as the service).
job "service" {
datacenters = ["dc1"]
type = "service"
group "service" {
network {
mode = "bridge"
# port "compact_thrift" {
# to = 6831
# static = 6831
# }
# port "frontend" {
# to = 16686
# }
}
# service {
# name = "tracing"
# port = "compact_thrift"
# connect {
# sidecar_service {}
# }
# }
# task "jaeger" {
# driver = "docker"
# env {
# SPAN_STORAGE_TYPE = "badger"
# }
# config {
# image = "jaegertracing/all-in-one:1.21.0"
# ports = ["compact_thrift", "frontend", "collector"]
# }
# }
service {
name = "service"
connect {
sidecar_service {
proxy {
upstreams {
destination_name = "tracing"
local_bind_port = 6831
}
}
}
}
}
task "service" {
driver = "docker"
config {
image = "python:3.9.1-buster"
command = "python3"
args = ["local/service.py"]
}
template {
destination = "local/service.py"
data = <<EOF
import os
os.system("pip3 install jaeger_client")
import logging
import time
from jaeger_client import Config
def main():
log_level = logging.DEBUG
logging.getLogger("").handlers = []
logging.basicConfig(format="%(asctime)s %(message)s", level=log_level)
config = Config(
config={
"sampler": {
"type": "const",
"param": 1,
},
"logging": True,
},
service_name="your-app-name",
validate=True,
)
tracer = config.initialize_tracer()
with tracer.start_span("TestSpan") as span:
span.log_kv({"event": "test message", "life": 42})
with tracer.start_span("ChildSpan", child_of=span) as child_span:
child_span.log_kv({"event": "down below"})
time.sleep(2)
tracer.close()
if __name__ == "__main__":
main()
# Send one trace then keep the container alive for debugging.
time.sleep(2000)
EOF
}
}
}
}