Distributed Tracing and Context Propagation
Concept
A single user request in a microservices architecture can fan out into a call chain touching a dozen services, and when that request is slow or errors out, the question "which of those dozen hops caused it" is unanswerable from any one service's logs alone. Distributed tracing solves this by recording the shape of the call chain itself, not just what each service did in isolation.
A trace represents one end-to-end request. It's composed of spans — each span is a single unit of work (a specific operation: an HTTP call, a database query, a function execution) with a start time, duration, and metadata. Spans are linked into a parent-child tree: the span for the top-level request is the root, and every downstream call it makes creates a child span, recursively, so the resulting trace is a timeline showing exactly which hop took how long and where an error was first raised.
The mechanism that makes this possible across process and network boundaries is context propagation. When service A calls service B, A doesn't just send the business payload — it also injects trace context (a trace ID identifying the overall request, and a span ID identifying the specific call) into the outgoing request, typically as HTTP headers. Service B extracts that context on receipt, creates its own span as a child of A's, and propagates the (now-updated) context onward to whatever it calls next. Without this propagation step, each service would generate its own disconnected trace with no way to stitch them into one causal picture — propagation is what turns "logs scattered across a dozen services" into "one coherent trace."
This approach traces back to Google's internal Dapper system, described in a 2010 paper, which established the trace/span model as the industry pattern. Multiple open-source implementations followed (Zipkin, Jaeger, OpenTracing, OpenCensus), which fragmented the ecosystem until OpenCensus and OpenTracing merged in 2019 to form OpenTelemetry — now the de facto standard for instrumentation, with the W3C Trace Context HTTP header format as its default propagation standard (older systems commonly used the B3 format instead, which OpenTelemetry still supports for interop).
Tradeoffs
| Approach | Benefit | Cost |
|---|---|---|
| No tracing (logs + timestamps only) | Zero instrumentation cost | Manually correlating logs by timestamp across services is slow and often ambiguous once concurrent requests interleave |
| Manual correlation ID only (no span tree) | Simple to add, ties logs together | Answers "which requests are related" but not "which specific hop was slow" — no timing/parent-child structure |
| Full distributed tracing (spans + propagation) | Exact call-chain timing and structure, pinpoints the failing/slow hop directly | Requires instrumentation at every service boundary, a collector/backend to store and query traces, and propagation to be wired through every outbound call — a single un-instrumented hop breaks the chain for everything downstream of it |
When to use / when not to
- Valuable once a request genuinely crosses more than one service boundary — a single monolith with no network hops has nothing for tracing to correlate; a correlation ID in structured logs already covers that case adequately.
- Especially valuable at fan-out points (an API gateway calling several backend services) or deep call chains, where the question "which of these N hops is the slow one" can't be answered any other way without stepping through logs by hand.
- Don't reach for a full tracing backend (Jaeger, etc.) before there's more than one hop to trace — the instrumentation and infrastructure cost isn't justified until the call chain actually spans processes.
- Sampling matters at scale: tracing every single request in a high-throughput system is often unnecessary and costly to store; head-based sampling (decide to trace at the start) or tail-based sampling (decide after the fact, keeping traces that errored or were slow) trade completeness for cost.
Common pitfall
Instrumenting most services but missing propagation through one hop — often an async boundary (a message queue, a background job, a webhook callback) where the trace context isn't naturally part of the request/response path the way an HTTP header is. The trace breaks at exactly that point: everything upstream looks fine, everything downstream of the gap starts a new, disconnected trace, and the operator debugging an incident sees two unrelated-looking traces instead of one continuous one, with no obvious indication that they were originally the same request. The fix is treating context propagation as a requirement at every boundary a request crosses, not just synchronous HTTP calls — async messaging systems need the trace context explicitly carried in the message envelope/attributes, propagated manually since there's no header to piggyback on.
Engineering Lens
The strong review answer isn't "we added tracing" as a checkbox — it's being able to name every boundary type a request crosses in the actual system (synchronous HTTP, async queue, scheduled job, webhook) and confirming context propagation is wired through each one, not just the synchronous HTTP calls that instrumentation libraries handle automatically out of the box. A trace that silently breaks at the one async hop nobody thought to check is worse than no tracing in one specific way: it creates false confidence that the observability gap has been closed, right up until an incident needs the broken link and it isn't there.