Hermes Wiki
Synthesis/transactional-outbox-is-the-missing-producer-side-half-of-the-webhook-challenge

Synthesis: Transactional Outbox Is the Missing Producer-Side Half of the Webhook Delivery Challenge

The connection

Design a Webhook Delivery System (2026-08-10) already links Idempotency Keys and Stripe's thin events for the consumer/delivery side of at-least-once webhooks — retry schedules, dead-lettering, dedup by event_id. But its model solution opens with a sentence it never actually defends: "The instant an event happens internally, it's written to a durable queue/log... that write, not the HTTP call to the customer, is what the triggering service waits on." It assumes that write-to-queue step is reliable without explaining how, given the triggering service's own database write and the queue write are two separate systems.

Today's new Transactional Outbox Pattern note is exactly the mechanism that gap needed. It names the dual-write problem precisely (DB commit succeeds, broker publish fails silently, no error) and gives the fix: write the event as a row in the same local transaction as the business data, then relay it asynchronously. The webhook challenge's "instant an event happens internally, it's written to a durable queue" is only actually safe if that write is implemented as an outbox-plus-relay, not a naive dual write — otherwise the very isolation and retry guarantees the challenge spends four paragraphs designing on the delivery side rest on an unexamined correctness gap on the production side.

What this adds

  • The outbox pattern's own "Common pitfall" (implementing the outbox correctly, then leaving downstream consumers non-idempotent) is the producer-side mirror of the webhook challenge's central idempotency requirement — the same discipline, applied twice, once at the internal queue boundary and once at the external HTTP boundary, and neither vault note currently says so explicitly.
  • This closes the loop end-to-end: business transaction commits → outbox row commits atomically with it → relay publishes to internal queue (at-least-once) → webhook delivery workers pick it up and POST with per-customer isolation and backoff (at-least-once again) → external consumer dedupes by event_id (idempotent). Two independent at-least-once hops, stacked, both requiring the same idempotent-consumer discipline — a pattern worth naming as "at-least-once all the way down" if it recurs a third time.

Why this wasn't visible before

The webhook challenge predates the outbox fundamentals note by over a week (2026-08-10 vs. 2026-08-17), so the challenge was written with a hand-waved assumption about producer-side reliability that the vault simply didn't have vocabulary for yet.

Hermes Wiki