Developer/Languages/Python/ErrorHandlingAndLogging/Fundamentals/exception-design-and-structured-logging-in-python

Exception Design and Structured Logging in Python

Concept

Two separate design decisions get conflated under "error handling": what exception types a codebase raises and catches, and what gets logged when something goes wrong. Getting either one wrong hides the information an on-call engineer needs during an incident.

Exception hierarchy design: catching bare Exception (or worse, bare except:) at a service boundary swallows programming errors (a TypeError from a bug) indistinguishably from expected failure modes (a ConnectionError from a downstream dependency being down). The fix is a small hierarchy of custom exceptions rooted in a package-specific base class (e.g. class PaymentError(Exception), with PaymentDeclinedError/PaymentTimeoutError subclasses), so a catch block can be specific about which failures it's actually prepared to handle and let anything else propagate — an unhandled exception that crashes loudly is more useful than one silently caught and papered over.

Logging: Python's standard logging module is a handler/formatter/level pipeline — a Logger emits LogRecords to one or more Handlers (console, file, network), each with its own Formatter and level filter. The critical API for exceptions specifically is logger.exception(msg) (equivalent to logger.error(msg, exc_info=True)), called from inside an except block — it attaches the full traceback to the log record automatically, which a plain logger.error(str(e)) does not. structlog builds on top of this model for structured (machine-parseable) logs: instead of formatting a human-readable string, it treats each log call as a dict of key-value pairs that flows through a chain of small processor functions before final rendering (as JSON, colored console output, etc.), and lets a logger accumulate bound context (log.bind(request_id=rid)) that automatically attaches to every subsequent call on that logger instance.

Tradeoffs

Approach	Benefit	Cost
Catch bare `Exception` everywhere	Never crashes on an unexpected error	Hides real bugs as "handled" errors; a `TypeError` from a typo gets treated the same as an expected timeout, and both get silently retried or logged at the wrong severity
Custom exception hierarchy, narrow `except` clauses	Callers can distinguish expected failure modes from bugs; unexpected exceptions propagate and surface loudly	More types to define and maintain; overzealous hierarchies (a new exception class per tiny variant) become their own maintenance burden
Plain `logging` module, string-formatted messages	Zero dependencies, built into the standard library, sufficient for small services	Free-text messages are hard to query/aggregate at scale ("find all failed payments for user X" means grepping strings, not filtering a field)
`structlog` (or equivalent structured logging)	Every log line is a queryable record with consistent fields; bound context (request ID, user ID) automatically threads through a whole request without manual re-passing	Extra dependency and a processor-chain concept to learn; needs discipline to keep field names consistent across the codebase or queries silently miss records using the wrong key

When to use / when not to

Define a custom exception hierarchy at any service or module boundary where callers need to distinguish "this failed in an expected way I should handle" from "this is a bug." A one-off internal helper function with a single obvious caller doesn't need its own exception type — that's over-engineering for no real benefit.
Reach for logger.exception() (not logger.error()) specifically inside except blocks — the difference is exactly whether the traceback shows up in the log, and losing it means recreating the failure from scratch during an incident instead of reading the log.
Adopt structured logging (structlog or a JSON formatter on stdlib logging) once logs are consumed by a log aggregation/observability platform (CloudWatch Logs Insights, Datadog, ELK) rather than read as a raw file — structured fields are what make querying and dashboarding possible; plain text logs make every query a regex.
Don't configure handlers/formatters at import time inside a reusable library module — a library should call logging.getLogger(__name__) and let it emit records, but leave handler/level configuration to the application's entry point; a library that configures logging globally on import breaks the application's own logging setup out from under it.

Common pitfall

Logging the exception's string representation instead of the exception object with traceback context — logger.error(f"Payment failed: {e}") records only the exception's message (e.g. "Connection refused"), discarding which line raised it, which function called that line, and the full call stack that got there. During an incident, the difference is being able to jump straight to the failing code path versus starting a fresh investigation from a one-line message with no stack trace. The standard-library fix is trivial (logger.exception(...) inside the except block, or exc_info=True outside one) but easy to miss because both versions look equally "handled" from the code's perspective — the log call itself doesn't error either way.

Engineering Lens

Exception design and logging design are really the same underlying decision made twice: what information does the next person (or the same person, three months later) need to diagnose this failure without re-deriving it from scratch? A narrow exception hierarchy answers that question at the code level — "was this expected or a bug" — while structured logging with full tracebacks answers it at the operational level — "what exactly happened, queryable across every instance of this failure." Both fail the same way when skipped: except Exception and logger.error(str(e)) are each individually reasonable-looking code, and the gap only becomes visible during the incident where the missing context would have mattered.