Architecture/Fundamentals/change-data-capture

Change Data Capture (CDC)

Concept

Systems constantly need one database's changes to reach somewhere else — a search index, a cache, a data warehouse, a downstream microservice, an analytics stream. The naive ways to do this are all flawed: dual writes (the app writes to the database and publishes an event) can leave the two out of sync when one succeeds and the other fails; polling ("select rows where updated_at > last_seen") misses deletes, misses intermediate states, adds query load, and lags behind. Change Data Capture solves this properly by treating the database's own change history as an event source.

CDC captures row-level changes (inserts, updates, deletes) from a datastore as they are committed and delivers them as an ordered stream of change events to consumers. The gold-standard implementation is log-based CDC: rather than querying tables, the CDC tool reads the database's transaction log — Postgres's WAL (write-ahead log), MySQL's binlog, MongoDB's oplog — which the database already writes for its own durability and replication. Because the transaction log is the authoritative, ordered record of every committed change, log-based CDC is:

Complete — it sees every change including deletes and every intermediate update, not just the latest state a poll would find.
Ordered — changes arrive in commit order, preserving causality.
Low-overhead — it reads a log the database is already producing, rather than adding query load to the tables.
Low-latency — changes propagate as they commit, not on a polling interval.

The most-cited tool is Debezium (streaming into Kafka), but the pattern is broader than any one product. CDC is the engine behind database replication, cache invalidation, keeping search indexes fresh, feeding data lakes/warehouses, and the transactional outbox (where the app writes an event row in the same transaction as its business change, and CDC ships that row — solving the dual-write problem for real).

Tradeoffs

Approach	Completeness	DB overhead	Latency	Ordering
Dual writes (app publishes event)	Fragile — can desync on partial failure	None extra	Low	App-controlled, easy to get wrong
Polling (`updated_at`)	Misses deletes & intermediate states	Adds query load	Bounded by poll interval	Approximate
Trigger-based CDC	Complete	High — triggers fire on every write, in the write path	Low	Good
Log-based CDC	Complete, including deletes	Low — reads existing log	Low	Strong (commit order)

Log-based CDC's costs are real but different in kind: it couples you to the database's internal log format and requires operational access to that log (replication slots, binlog retention config), it needs care around schema changes (a column added upstream must be handled downstream — see Pinterest's automated schema evolution), and consumers must be idempotent because at-least-once delivery means change events can be redelivered. There's also an initial snapshot problem: a new consumer needs the current full state before it can meaningfully apply the ongoing change stream, so CDC tools bootstrap with a snapshot then switch to streaming from the log.

When to use / when not to

Use to keep derived data stores in sync with a source of truth — search indexes, caches, read models, materialized views, data warehouses — without dual-write fragility.
Use to build event-driven integration off an existing database without rewriting the application to publish events itself (a pragmatic path for legacy systems that can't easily emit domain events).
Use with the transactional outbox to reliably publish domain events: write the event to an outbox table in the same DB transaction as the business change, and let CDC stream the outbox — the event and the state change commit atomically.
Prefer log-based over trigger-based or polling whenever you have access to the transaction log; fall back to triggers/polling only when the log is inaccessible (some managed DBs, some legacy engines).
Don't use CDC as a substitute for a real event/domain model when you're building a new system that can emit meaningful business events directly — raw row-change events leak the internal schema to consumers and couple them to your table structure.

Common pitfall

Leaking the source database's schema to every downstream consumer. Raw CDC events are row changes — column names, types, table structure — so consumers end up coupled to the upstream schema, and an innocuous upstream migration (renaming a column, splitting a table) breaks every consumer at once. The mitigation is a transformation/contract layer (or the outbox pattern, which lets you publish a stable event shape instead of raw rows) plus deliberate schema-evolution handling. The second classic pitfall is assuming exactly-once, ordered, forever-retained delivery: CDC is at-least-once (consumers must dedupe), transaction logs have finite retention (a consumer offline too long can miss changes that aged out of the log), and re-snapshots are needed after such gaps.

Principal Engineer Lens

CDC is the concrete answer to a question that comes up in nearly every data-integration review: "how does this change get to that system without dual-write bugs?" The Principal-level insight is recognizing that the database's transaction log is already a perfect, ordered event stream the database maintains for its own replication — CDC just taps it, turning the log from an internal durability mechanism into an integration backbone. That reframing (log-as-source-of-truth, everything else as a materialized view of the log) is the same idea underneath event sourcing, Kafka's design, and database replication itself. In Fintech and data-heavy platforms it's also the workhorse for feeding audit systems, reconciliation pipelines, and regulatory data lakes off a transactional source of truth without adding load to the transactional path — and the reviewable nuances (idempotent consumers, schema-evolution strategy, log retention vs. consumer lag) are exactly what separates a design that works in a demo from one that survives a consumer being down for six hours.

Reel Script

Setup: Your orders live in Postgres. Your search index, your cache, and your data warehouse all need to reflect every order change. The tempting fix — have the app write to Postgres and publish an event — quietly breaks the day one write succeeds and the other fails, and now your search index disagrees with your database.

Concept walkthrough: Change Data Capture reads the database's own transaction log — the WAL in Postgres, the binlog in MySQL — which the database already writes for durability and replication. Every committed insert, update, and delete is in that log, in commit order. CDC turns that log into an ordered stream of change events that any number of consumers can subscribe to. No dual writes, no polling, no missed deletes.

Real example tie-in: Walk the transactional outbox: the app writes its business change and an event row in the same transaction, so they commit together atomically — then CDC streams the outbox table. The event and the state change can never disagree, because they were one transaction. That's how you publish reliable domain events from a database.

Tradeoffs & alternatives: Contrast with polling (misses deletes and intermediate states, adds load) and triggers (complete but heavy, in the write path). Log-based CDC is complete and low-overhead but couples you to the log format, needs idempotent consumers (delivery is at-least-once), and needs a schema-evolution plan — a renamed upstream column can break every consumer.

Principal Engineer takeaway: The insight is that the transaction log is already a perfect ordered event stream — CDC just taps it. Treat the log as the source of truth and everything downstream as a materialized view of it. In review, the answer to "how does this data get there reliably" is CDC off the log, plus idempotent consumers and a schema-evolution strategy — not app-level dual writes.

Sources: