Design a Webhook Delivery System
Scenario prompt
You're building the outbound webhook system for a platform (think Stripe, GitHub, or Shopify) that notifies thousands of third-party customer endpoints whenever an event happens (a payment succeeds, an order ships, a build finishes). Requirements:
- Customer endpoints are unreliable — slow, occasionally down, sometimes misconfigured — and none of that should back up event processing for everyone else
- Deliveries must not be lost even if a customer's endpoint is down for hours
- Customers must never process the same business event twice as a correctness bug, even though at-least-once delivery means the same HTTP POST can legitimately arrive more than once
- Any customer must be able to prove a webhook actually came from you, not an attacker who guessed their endpoint URL
Mihir's attempt
[!todo] Write your own attempt here before reading the model solution below — how you'd isolate one customer's slow endpoint from every other customer's deliveries, and how you'd let receivers de-duplicate safely.
Model solution
Decouple event production from delivery with a durable queue, and give every customer their own retry lane. The instant an event happens internally, it's written to a durable queue/log (Kafka, SQS, etc.) — that write, not the HTTP call to the customer, is what the triggering service waits on. A pool of delivery workers then drains the queue and attempts the actual POST. Critically, per-customer (or per-endpoint) partitioning — a dedicated queue, or at minimum a per-endpoint concurrency limit and backoff state — means one customer's dead or crawling endpoint only slows down that customer's deliveries, never the shared worker pool. Without this isolation, a single slow endpoint can starve delivery capacity for every other customer, the classic noisy-neighbor failure this system exists to prevent.
Retry with exponential backoff and a bounded schedule, then park failures for human/customer visibility. A failed delivery (non-2xx response, timeout, connection refused) gets retried on a backoff schedule (seconds, then minutes, then hours) capped at some maximum — commonly 24-72 hours of attempts before giving up. Every retry is a fresh delivery attempt of the same event, which is exactly why idempotency has to be solved on the receiving end rather than assumed away. Exhausted deliveries land in a dead-letter state surfaced to the customer (a dashboard, a digest email) rather than silently vanishing — an invisible permanent failure is worse than a visible one, per the same failure-transparency instinct behind Circuit Breaker Pattern.
Give every event a stable, unique ID and document it as the de-dup key. At-least-once delivery is a deliberate tradeoff — building exactly-once delivery over an unreliable network is prohibitively expensive, per the same "acknowledge to yourself, let the receiver dedupe" reasoning as Idempotency Keys. Every event carries an immutable event_id in its payload and a matching header, generated once at production time and preserved across every retry of that same delivery. The contract you publish to customers is explicit: "you may receive this event more than once; store event_ids you've already processed and skip duplicates." Pushing that burden onto documented behavior rather than pretending it doesn't exist is what makes the at-least-once tradeoff safe to ship.
Sign every payload (HMAC) so receivers can verify authenticity, and version the signing scheme. Each webhook carries a signature header — an HMAC-SHA256 of the raw payload body keyed on a per-customer shared secret — that the receiver recomputes and compares before trusting the payload, the same defense-in-depth instinct as Defense in Depth applied to inbound trust rather than infrastructure layers. Timestamping the payload and rejecting stale signatures blocks replay of a captured request. Versioning the signing scheme (a v1=, v2= prefix) from day one avoids a painful flag-day migration later, mirroring the version-stability lesson in Stripe: Thin Events and Notification Handlers for Version-Stable Webhooks.
Gaps to revisit
- Payload versioning and schema evolution — how do you add fields to an event without breaking customers who parse it strictly, and is a "thin event + fetch the object" model (as Stripe adopted) worth the extra API round-trip it imposes on receivers?
- Ordering guarantees — do customers need events for the same resource delivered in order, and what does per-endpoint partitioning cost you if so?
- Self-service replay — letting a customer manually re-trigger delivery of a specific past event after fixing their endpoint, without re-emitting it as a "new" occurrence internally
Engineering Lens
The core tension is isolation versus efficiency: a shared delivery pool is cheaper to run than one lane per customer, but it's also one slow customer away from degrading everyone. Naming that tradeoff — and defending where you draw the isolation boundary (per-customer queue vs. per-customer concurrency cap vs. fully shared pool with backoff) — is the kind of blast-radius reasoning a Principal-level design review expects. It's also a direct analog to any fan-out-to-untrusted-third-parties problem: partner integrations, outbound notification systems, even multi-tenant SaaS callback hooks all inherit this same noisy-neighbor risk and the same at-least-once-plus-idempotency-contract fix.