Three Pillars of Observability
Concept
Observability is usually broken into three complementary data types, each answering a different question about a running system:
- Metrics — numerical measurements over time (CPU usage, request latency, error rate). Lightweight, cheap to aggregate and store long-term, ideal for dashboards and threshold-based alerting. Metrics tell you that something is wrong.
- Logs — immutable, timestamped records of discrete events, structured or unstructured. The detailed diary of what actually happened inside a component. Logs tell you the specific detail of what happened at a point in time.
- Traces — the end-to-end path of a single request as it flows across services, showing where time was spent and which downstream call failed. Traces tell you where in the system the problem occurred, across service boundaries a single log or metric can't see.
The common framing: metrics alert you to a problem, traces show you the request's path through the system to localize where it happened, and logs give you the granular context needed to actually resolve it. None of the three alone is sufficient — metrics without traces tell you something's slow but not where; traces without logs show you which hop failed but not why.
Tradeoffs
| Signal | Cost to collect/store | Cardinality tolerance | Best answers |
|---|---|---|---|
| Metrics | Cheap (pre-aggregated) | Low — high-cardinality labels (e.g., per-user) blow up storage/cost | "Is something wrong, and since when?" |
| Logs | Expensive at high volume (raw, often unstructured) | High — can capture arbitrary detail | "What exactly happened on this one instance/request?" |
| Traces | Moderate-to-expensive (per-request overhead, typically sampled) | High for the sampled subset | "Where in a multi-service call chain did the time go / did it fail?" |
Full-fidelity collection of all three, on every request, at scale, is prohibitively expensive — production systems make deliberate sampling and retention tradeoffs (e.g., trace 1% of requests fully, but always trace/log the ones that errored or exceeded a latency threshold).
When to use / when not to
- Metrics are the right default for anything you want to alert on or watch trend over time — they scale to production volume cheaply.
- Logs are essential at debug time but shouldn't be the primary alerting mechanism — grepping logs to detect an outage is slower and more expensive than a metric threshold catching it first.
- Traces earn their overhead specifically in systems with more than one service in the call path — a monolith with no network hops between components gets little value from distributed tracing, since a stack trace + logs already localize the problem.
- Under-investing in any one pillar creates a specific blind spot: metrics-only means you know something broke but not why; logs-only doesn't scale to alerting at volume; no tracing means multi-service latency problems get diagnosed by guesswork instead of data.
Common pitfall
Treating the three pillars as separate tools bought from separate vendors with no correlation ID tying them together. If a metric spike, the relevant trace, and the relevant log lines can't be pivoted between using a shared identifier (request ID, trace ID), the "three pillars" become three separate haystacks instead of one coherent debugging path — the value of observability is largely in the correlation, not just the collection.
Engineering Lens
The Principal-level framing of observability isn't "we have Datadog/Grafana dashboards" — it's being able to describe the actual debugging path from alert to root cause: which metric would fire first, which trace would show where the failure localized, and which log lines would explain why. Being able to walk that path concretely, for a specific incident class, before it happens, is what separates "we have monitoring" from "we can actually operate this system in production." This scales cleanly from a Network operations context (device health metrics, syslogs, flow traces) to a Fintech transaction pipeline (payment success-rate metrics, ledger logs, distributed traces across the settlement chain).