Three Pillars of Observability
Concept
Observability is usually broken into three complementary data types, each answering a different question about a running system:
- Metrics — numerical measurements over time (CPU usage, request latency, error rate). Lightweight, cheap to aggregate and store long-term, ideal for dashboards and threshold-based alerting. Metrics tell you that something is wrong.
- Logs — immutable, timestamped records of discrete events, structured or unstructured. The detailed diary of what actually happened inside a component. Logs tell you the specific detail of what happened at a point in time.
- Traces — the end-to-end path of a single request as it flows across services, showing where time was spent and which downstream call failed. Traces tell you where in the system the problem occurred, across service boundaries a single log or metric can't see.
The common framing: metrics alert you to a problem, traces show you the request's path through the system to localize where it happened, and logs give you the granular context needed to actually resolve it. None of the three alone is sufficient — metrics without traces tell you something's slow but not where; traces without logs show you which hop failed but not why.
Tradeoffs
| Signal | Cost to collect/store | Cardinality tolerance | Best answers |
|---|---|---|---|
| Metrics | Cheap (pre-aggregated) | Low — high-cardinality labels (e.g., per-user) blow up storage/cost | "Is something wrong, and since when?" |
| Logs | Expensive at high volume (raw, often unstructured) | High — can capture arbitrary detail | "What exactly happened on this one instance/request?" |
| Traces | Moderate-to-expensive (per-request overhead, typically sampled) | High for the sampled subset | "Where in a multi-service call chain did the time go / did it fail?" |
Full-fidelity collection of all three, on every request, at scale, is prohibitively expensive — production systems make deliberate sampling and retention tradeoffs (e.g., trace 1% of requests fully, but always trace/log the ones that errored or exceeded a latency threshold).
When to use / when not to
- Metrics are the right default for anything you want to alert on or watch trend over time — they scale to production volume cheaply.
- Logs are essential at debug time but shouldn't be the primary alerting mechanism — grepping logs to detect an outage is slower and more expensive than a metric threshold catching it first.
- Traces earn their overhead specifically in systems with more than one service in the call path — a monolith with no network hops between components gets little value from distributed tracing, since a stack trace + logs already localize the problem.
- Under-investing in any one pillar creates a specific blind spot: metrics-only means you know something broke but not why; logs-only doesn't scale to alerting at volume; no tracing means multi-service latency problems get diagnosed by guesswork instead of data.
Common pitfall
Treating the three pillars as separate tools bought from separate vendors with no correlation ID tying them together. If a metric spike, the relevant trace, and the relevant log lines can't be pivoted between using a shared identifier (request ID, trace ID), the "three pillars" become three separate haystacks instead of one coherent debugging path — the value of observability is largely in the correlation, not just the collection.
Principal Engineer Lens
The Principal-level framing of observability isn't "we have Datadog/Grafana dashboards" — it's being able to describe the actual debugging path from alert to root cause: which metric would fire first, which trace would show where the failure localized, and which log lines would explain why. Being able to walk that path concretely, for a specific incident class, before it happens, is what separates "we have monitoring" from "we can actually operate this system in production." This scales cleanly from a Network operations context (device health metrics, syslogs, flow traces) to a Fintech transaction pipeline (payment success-rate metrics, ledger logs, distributed traces across the settlement chain).
Reel Script
Setup: Say a dashboard shows p99 latency spiked five minutes ago. That metric alone tells you that something's wrong — it tells you nothing about why, or where in a ten-service call chain the time actually went.
Concept walkthrough: Walk through the three pillars in the order you'd actually use them during an incident — the metric alerts you first (cheap, fast, always-on), the trace localizes which service in the chain is slow or failing, and the logs from that specific service give you the granular detail (a specific exception, a specific query) that explains why.
Real example tie-in: Walk a concrete incident: checkout latency spikes (metric fires), the trace shows the time is concentrated in a call to the inventory service, and the inventory service's logs show a specific database connection pool exhaustion error — three pillars, three questions answered, one root cause.
Tradeoffs & alternatives: Note the cost reality — you can't full-fidelity trace and log every request at scale, so production systems sample traces and often only fully log the requests that errored or ran slow, using metrics as the cheap always-on layer that decides what's worth a closer look.
Principal Engineer takeaway: The tell in a review isn't listing the tools you use — it's being able to walk the exact alert-to-root-cause path for a specific failure mode before it happens, and pointing out that the real value is in the shared correlation ID tying metrics, traces, and logs together, not just having all three collected separately.
Related
Sources: