Transactional Outbox Pattern
Concept
A service that updates its own database and then publishes an event about that update to a message broker has to make two separate writes — one to the database, one to the broker — and those two writes cannot be wrapped in a single atomic transaction across two different systems. This is the dual-write problem: the database commit can succeed while the broker publish fails (or vice versa), and there's no way to guarantee both happen or neither does. Concretely, if the DB commit succeeds but the publish crashes before confirming, the event is silently lost forever and every downstream consumer that should have reacted to it never does — with no error, no retry, nothing to indicate the gap exists.
The Transactional Outbox pattern sidesteps the dual-write problem by not doing two writes to two systems at all. Instead, the service writes the event as a row in an outbox table, in the same local database transaction as the business data change — so both either commit together or roll back together, which is a guarantee a single database can actually make. A separate, asynchronous message relay process then reads unpublished rows from the outbox table and forwards them to the actual message broker, marking (or deleting) them once publish is confirmed. Two implementations of the relay are common: polling publisher, where the relay periodically queries the outbox table for new rows (simple, but adds polling latency and DB load); and transaction log tailing (also called change data capture, e.g. Debezium reading MySQL's binlog or Postgres's WAL), where the relay reads the database's own replication log instead of querying the table directly — lower latency, no polling load, but a heavier piece of infrastructure to run.
The relay itself can't offer perfect exactly-once delivery either — it can crash after publishing but before marking the outbox row as sent, and on restart it republishes that row. This means the pattern only fully closes the reliability gap when paired with consumer-side idempotency: every downstream consumer must be able to safely process the same event twice without double-applying its effect, the same discipline covered in Idempotency Keys. The outbox pattern guarantees at-least-once delivery of every event that was ever committed — it deliberately doesn't try to guarantee exactly-once, because that guarantee is generally unachievable across two independent systems without consumer-side cooperation.
Tradeoffs
| Approach | Consistency guarantee | Latency | Operational cost |
|---|---|---|---|
| Direct dual write (DB commit, then publish to broker) | None — either write can fail independently, silently losing events | Lowest | Lowest to build, but carries a real correctness gap |
| Two-phase commit (2PC) across DB and broker | Strong, atomic across both systems | Highest — blocks on a distributed transaction coordinator | High, and most modern message brokers don't support 2PC well or at all |
| Transactional outbox + polling relay | Effectively atomic at the DB layer; at-least-once delivery to the broker | Bounded by polling interval (typically sub-second to a few seconds) | Moderate — an extra table, a relay process, and a cleanup/archival job for published rows |
| Transactional outbox + log-tailing relay (CDC) | Same guarantee, tighter latency | Near-real-time — reacts to log writes directly | Higher — running a CDC pipeline (e.g. Debezium + Kafka Connect) is real infrastructure to operate |
The pattern trades a small amount of added latency and operational surface (an outbox table, a relay process) for closing a correctness gap that direct dual-writes simply cannot close — and it explicitly declines to solve the harder problem of exactly-once delivery, pushing that responsibility to idempotent consumers instead, which is usually the more tractable place to solve it.
When to use / when not to
- Use it any time a service must reliably publish an event as a direct consequence of a database write it just made — order placed, payment captured, inventory reserved — and losing that event silently would be a real correctness problem, not just a minor inconvenience.
- Pair it with the Saga pattern for distributed transactions — sagas are typically implemented as a chain of events, and the outbox pattern is what makes each step's event-publish step itself reliable.
- Prefer CDC/log-tailing relays over polling once event latency matters (near-real-time downstream reactions) or once outbox table polling starts adding meaningful load to the primary database.
- Don't reach for it when the "event" being published isn't tied to a specific local database transaction — e.g., publishing a scheduled/timer-based event has no dual-write problem to solve in the first place.
- Don't skip consumer-side idempotency and assume the outbox alone gives exactly-once delivery — it doesn't, and treating it as if it does is exactly the pitfall below.
Common pitfall
Implementing the outbox table and relay correctly, then assuming the reliability problem is fully solved — and leaving downstream consumers non-idempotent. The relay's own at-least-once semantics mean duplicate delivery is a normal, expected occurrence (not a rare edge case), and a non-idempotent consumer that double-applies a "payment captured" event will double-charge or double-fulfill. The outbox pattern only delivers its full guarantee as a system — reliable publish plus idempotent consumption — not as an isolated piece.
Engineering Lens
The transactional outbox pattern is a good test of whether someone actually understands the boundary of what a single database transaction can and can't guarantee — a lot of engineers reach for it by pattern-matching ("event-driven microservices need an outbox") without being able to explain precisely why a direct dual write is unsafe in the first place. In a design review, the sharp version of this conversation isn't "are we using an outbox" — it's "which specific write-then-publish paths in this system have a correctness requirement that events aren't silently dropped, and which relay mechanism fits their latency needs." For anything touching money movement or state that downstream systems act on irreversibly (a trade execution, a payment capture, a fund transfer), being able to point to the outbox-plus-idempotent-consumer combination as the closed-loop guarantee — not just "we have an outbox table" — is what reads as understanding the actual failure mode being defended against.