Design an Observability Pipeline for a Microservices Platform
Scenario prompt
A platform has grown from a monolith into ~80 microservices, owned by a dozen different teams. On-call engineers routinely spend 30-60 minutes just figuring out which service is actually responsible for a customer-facing error before they can even start debugging it. Design an observability pipeline that:
- Lets an on-call engineer go from "customers are seeing 500s" to "this specific service and deploy" in minutes, not the better part of an hour
- Correlates logs, metrics, and traces across service boundaries for a single request
- Scales cost-effectively as service count and traffic keep growing (raw log/trace volume can't just be retained forever at full fidelity)
- Doesn't require every team to hand-instrument their service from scratch
Mihir's attempt
[!todo] Write your own attempt here before reading the model solution below — what would you standardize, what would you sample vs. keep in full, and how would you make cross-service correlation actually work.
Model solution
The three pillars, tied together by one shared identifier. Logs, metrics, and traces each answer a different question (see Three Pillars of Observability) — traces show where time went across services, metrics show whether something's wrong in aggregate, logs show why for a specific request. None of them is sufficient alone, and none of them is useful across a microservices boundary unless every service propagates the same trace/correlation ID through every inbound and outbound call. This is the single highest-leverage decision in the whole design: without a shared ID threaded through every hop, "correlate logs and traces across service boundaries" is simply not possible after the fact, no matter how much data you collect.
Standardize instrumentation via a shared library/sidecar, not a per-team mandate. Asking a dozen teams to each hand-roll trace propagation, structured logging, and metric emission produces a dozen slightly-incompatible implementations. Instead, ship a thin shared library (or a sidecar/service-mesh layer like Envoy/Istio) that auto-instruments HTTP/gRPC calls, injects the correlation ID, and emits traces/metrics in a standard format (OpenTelemetry is the pragmatic default — vendor-neutral, wide language support). Teams get useful observability by default just from adopting the library; opting into deeper custom spans is additive, not a hard requirement to get started.
Sample traces, don't sample metrics or error logs. At 80 services and real production traffic, full-fidelity trace retention for every request is both expensive and mostly wasted — the overwhelming majority of traces look identical and boring. Head-based sampling (keep 1% of requests, decided at the start of the trace) is simple but can miss the specific slow/erroring request an engineer actually needs. Tail-based sampling (buffer the trace, decide after seeing the outcome — keep 100% of errors and slow requests, sample down the fast/successful ones) costs more infrastructure but directly solves the on-call engineer's actual problem: the traces that matter are exactly the ones most likely to be kept. Metrics and error-level logs, by contrast, should not be sampled — they're the aggregate signal and the specific failure detail respectively, and both are cheap relative to full trace volume.
One pane of glass for correlation, not a per-team dashboard sprawl. A central observability platform (whether self-hosted Grafana/Loki/Tempo/Prometheus or a vendor like Datadog/Honeycomb) that lets an engineer pivot from a metric spike straight into the traces and logs for that time window, filtered by the correlation ID, is what actually collapses the 30-60 minute triage time. Per-team siloed tooling recreates the original problem — you still have to know which team's dashboard to check first.
Gaps to revisit
- How do you keep instrumentation-library adoption from silently drifting out of date across a dozen independently-deployed teams — a shared library only helps if teams actually stay current on it?
- What's the actual cost model for tail-based sampling at this scale — the trade sampling buys is engineer-hours saved vs. infrastructure spend for buffering/collector capacity, and that crossover point matters for justifying the design.
- How does this pipeline handle a cascading failure where the observability pipeline itself is degraded by the same incident it's supposed to help diagnose (e.g., the shared correlation-ID service or trace collector is down)?
Principal Engineer Lens
This challenge is really about a much broader principle: distributed-systems debuggability is a design decision made before the incident, not a tool bolted on during one. An architecture review that asks "how will an on-call engineer actually find the root cause at 3am" during design time — rather than after the first bad incident — is exactly the kind of forward-looking tradeoff framing that reads as Principal-level thinking. It also directly informs a cost pillar tradeoff (full trace retention is expensive) against operational excellence (fast triage saves engineer-hours and reduces customer-facing downtime), which is the kind of cross-pillar reasoning that shows up constantly in real architecture reviews regardless of industry — a trading platform's incident response and a payments platform's on-call rotation face the identical version of this problem.
Reel Script
Setup: Imagine you've grown from one monolith to 80 microservices owned by a dozen teams, and now every incident starts with 30-60 minutes of just figuring out which service is even responsible before anyone can start debugging.
Concept walkthrough: Walk through why logs, metrics, and traces are each necessary but individually insufficient, then land on the real fix: a shared correlation ID propagated through every service-to-service call, so a single request can be reconstructed end-to-end no matter how many services it touched.
Real example tie-in: Trace through a concrete incident — a spike in 500s on the customer-facing API. Show how a shared trace ID lets the on-call engineer pivot from the metric spike straight into the exact traces and logs for that time window, instead of guessing which of 80 services to check first.
Tradeoffs & alternatives: Contrast head-based sampling (cheap, but might drop exactly the slow/failing trace you need) against tail-based sampling (keeps 100% of errors and slow requests, costs more to buffer). Also touch on shared-library instrumentation vs. asking every team to hand-roll their own — consistency vs. team autonomy.
Principal Engineer takeaway: Debuggability at scale is a design-time decision, not something you retrofit after a bad incident — and the same reasoning about correlation IDs, sampling cost, and single-pane-of-glass triage applies whether you're running a microservices platform, a trading system, or a payments pipeline.