Design a Webhook Delivery System
Scenario prompt
You're building the outbound webhook system for a platform (think Stripe, GitHub, or Shopify) that notifies thousands of third-party customer endpoints whenever an event happens (a payment succeeds, an order ships, a build finishes). Requirements:
- Customer endpoints are unreliable — slow, occasionally down, sometimes misconfigured — and none of that should back up event processing for everyone else
- Deliveries must not be lost even if a customer's endpoint is down for hours
- Customers must never process the same business event twice as a correctness bug, even though at-least-once delivery means the same HTTP POST can legitimately arrive more than once
- Any customer must be able to prove a webhook actually came from you, not an attacker who guessed their endpoint URL
Mihir's attempt
[!todo] Write your own attempt here before reading the model solution below — how you'd isolate one customer's slow endpoint from every other customer's deliveries, and how you'd let receivers de-duplicate safely.
Model solution
Decouple event production from delivery with a durable queue, and give every customer their own retry lane. The instant an event happens internally, it's written to a durable queue/log (Kafka, SQS, etc.) — that write, not the HTTP call to the customer, is what the triggering service waits on. A pool of delivery workers then drains the queue and attempts the actual POST. Critically, per-customer (or per-endpoint) partitioning — a dedicated queue, or at minimum a per-endpoint concurrency limit and backoff state — means one customer's dead or crawling endpoint only slows down that customer's deliveries, never the shared worker pool. Without this isolation, a single slow endpoint can starve delivery capacity for every other customer, the classic noisy-neighbor failure this system exists to prevent.
Retry with exponential backoff and a bounded schedule, then park failures for human/customer visibility. A failed delivery (non-2xx response, timeout, connection refused) gets retried on a backoff schedule (seconds, then minutes, then hours) capped at some maximum — commonly 24-72 hours of attempts before giving up. Every retry is a fresh delivery attempt of the same event, which is exactly why idempotency has to be solved on the receiving end rather than assumed away. Exhausted deliveries land in a dead-letter state surfaced to the customer (a dashboard, a digest email) rather than silently vanishing — an invisible permanent failure is worse than a visible one, per the same failure-transparency instinct behind Circuit Breaker Pattern.
Give every event a stable, unique ID and document it as the de-dup key. At-least-once delivery is a deliberate tradeoff — building exactly-once delivery over an unreliable network is prohibitively expensive, per the same "acknowledge to yourself, let the receiver dedupe" reasoning as Idempotency Keys. Every event carries an immutable event_id in its payload and a matching header, generated once at production time and preserved across every retry of that same delivery. The contract you publish to customers is explicit: "you may receive this event more than once; store event_ids you've already processed and skip duplicates." Pushing that burden onto documented behavior rather than pretending it doesn't exist is what makes the at-least-once tradeoff safe to ship.
Sign every payload (HMAC) so receivers can verify authenticity, and version the signing scheme. Each webhook carries a signature header — an HMAC-SHA256 of the raw payload body keyed on a per-customer shared secret — that the receiver recomputes and compares before trusting the payload, the same defense-in-depth instinct as Defense in Depth applied to inbound trust rather than infrastructure layers. Timestamping the payload and rejecting stale signatures blocks replay of a captured request. Versioning the signing scheme (a v1=, v2= prefix) from day one avoids a painful flag-day migration later, mirroring the version-stability lesson in Stripe: Thin Events and Notification Handlers for Version-Stable Webhooks.
Gaps to revisit
- Payload versioning and schema evolution — how do you add fields to an event without breaking customers who parse it strictly, and is a "thin event + fetch the object" model (as Stripe adopted) worth the extra API round-trip it imposes on receivers?
- Ordering guarantees — do customers need events for the same resource delivered in order, and what does per-endpoint partitioning cost you if so?
- Self-service replay — letting a customer manually re-trigger delivery of a specific past event after fixing their endpoint, without re-emitting it as a "new" occurrence internally
Principal Engineer Lens
The core tension is isolation versus efficiency: a shared delivery pool is cheaper to run than one lane per customer, but it's also one slow customer away from degrading everyone. Naming that tradeoff — and defending where you draw the isolation boundary (per-customer queue vs. per-customer concurrency cap vs. fully shared pool with backoff) — is the kind of blast-radius reasoning a Principal-level design review expects. It's also a direct analog to any fan-out-to-untrusted-third-parties problem: partner integrations, outbound notification systems, even multi-tenant SaaS callback hooks all inherit this same noisy-neighbor risk and the same at-least-once-plus-idempotency-contract fix.
Reel Script
Setup: You're sending event notifications to thousands of third-party endpoints you don't control — some fast, some slow, some down. How do you keep one dead endpoint from backing up notifications for everyone else, and how do receivers safely handle getting the same notification twice?
Concept walkthrough: Explain the durable-queue-plus-worker-pool architecture, why per-customer isolation (dedicated lanes or concurrency caps) is the load-bearing design decision, and the exponential-backoff-then-dead-letter retry lifecycle.
Real example tie-in: Walk through the Stripe thin-events case study — why they moved toward small, stable notification payloads plus a fetch-the-object-if-you-need-detail model, and how that sidesteps a whole class of payload-versioning breakage that a fat-payload webhook design runs into.
Tradeoffs & alternatives: Contrast at-least-once-plus-idempotency-key (cheap, standard, requires receiver cooperation) against building exactly-once delivery yourself (expensive, still imperfect) — and why the industry converged on pushing dedup onto the receiver rather than solving it centrally.
Principal Engineer takeaway: Any time you fan out to endpoints you don't control, isolation-per-recipient and an explicit at-least-once contract are the two decisions that determine whether the system degrades gracefully or cascades — naming both explicitly is what separates a working webhook system from one that just happens to work today.