Developer/Observability/Logging/Fundamentals/structured-logging

Structured Logging

Concept

A plain-text log line like User 4821 failed login from 10.0.4.2 (bad password) is easy for a human to read once, but nearly impossible for a machine to reliably parse across thousands of variations of the same message written by different engineers over years. Structured logging replaces the free-text sentence with a fixed set of key-value fields — almost always serialized as JSON — so every log entry is a machine-readable record from the moment it's written, not text that has to be regex-scraped after the fact: {"event":"login_failed","user_id":4821,"src_ip":"10.0.4.2","reason":"bad_password","ts":"2026-08-31T09:14:02Z"}.

The shift matters because production debugging is almost always a filter-and-correlate problem — "show me every failed login for this user in the last hour" or "show me every request on this trace ID across all four services it touched" — and that only works cheaply if user_id and trace_id are queryable fields, not substrings buried in prose that differs slightly every time. A logging library (not a hand-rolled print/console.log) is the mechanism: it owns serialization, guarantees a timestamp and severity level are always present, and gives every call site a consistent field vocabulary instead of each developer inventing their own message shape.

The fields that matter most across virtually every log entry: a timestamp, a severity level, a service/environment name, a request_id or trace_id for correlation, and whatever domain-specific fields the event needs (user_id, order_id, http_status). Consistency in field names across the whole codebase (user_id everywhere, never userId in one service and uid in another) is what makes cross-service queries and dashboards work at all.

Tradeoffs

Approach	Human readability (raw)	Machine queryability	Cost
Unstructured free-text	High — reads naturally in a terminal	Low — requires regex/grok parsing that breaks when message wording changes	Cheapest to write casually, most expensive to query later
Structured (JSON) logging	Low raw (needs a viewer/formatter)	High — every field is directly filterable/aggregable	Slightly more storage (structured JSON typically runs 1.5-2x the bytes of the equivalent plain-text line before compression), and requires a logging library/convention up front
Structured + log aggregation platform (ELK, Datadog, etc.)	High (platform renders it readably)	Highest — full-text search, field filters, and dashboards over the same data	Adds an operational dependency (the aggregation pipeline itself) and its own cost curve at high volume

The real tension isn't "JSON vs plain text" in isolation — it's that the value of structure only shows up later, during an incident, while the cost (a bit more storage, a bit more upfront discipline) is paid on every single line written. Teams that skip structured logging early almost always regret it the first time they need to correlate a single request across services during an outage and have nothing but inconsistent prose to grep through.

When to use / when not to

Use structured logging by default in any service that will run in production and be operated by more than one person — the payoff (fast incident correlation) compounds with team size and system complexity.
Always include a correlation ID (request_id or trace_id) that's generated or propagated at the edge and threaded through every downstream call — without it, structured fields still can't answer "what happened to this one request across four services."
Skip the overhead for a genuinely disposable script or local dev-only tool where no one will ever query the output programmatically — plain print debugging is fine there.
Don't log secrets, tokens, passwords, or full PII fields into structured logs "because it's just a field now" — a queryable field is easier to accidentally expose in a dashboard or export than the same value buried in unstructured prose, not harder.

Common pitfall

Treating structured logging as "wrap the same free-text message in a JSON envelope" instead of actually decomposing it into fields — e.g. {"message": "User 4821 failed login from 10.0.4.2 (bad password)"} is still JSON, but it gives up almost all of the actual benefit, since user_id and src_ip aren't independently filterable fields, just substrings of one string field. The discipline that matters is pulling every piece of variable data out into its own named key, not the serialization format itself.

Engineering Lens

The strong signal in a design review isn't "we use structured logging" as a checkbox — it's whether the field vocabulary is actually consistent across services, because a logging convention that only one team follows still leaves cross-service incident correlation broken in exactly the way structured logging was supposed to fix. The harder, more valuable conversation is: who owns the shared field schema (user_id, trace_id, tenant_id) across a multi-service org, and what enforces it — a shared logging library, a linter, or just tribal knowledge that erodes the moment a new team spins up a service. That governance question is domain-independent: a fintech's payment-processing pipeline and a media company's content-ingestion pipeline both fail the same way at 3am if order_id is spelled three different ways across the services a request actually touched.