Developer/CommunicationPatterns/Webhooks/Fundamentals/webhook-delivery-reliability

Webhook Delivery Reliability

Concept

A webhook inverts the usual client-server relationship: instead of your service polling a third party for changes, the third party POSTs an event to a URL you registered the moment something happens. That inversion buys real latency and efficiency wins over polling, but it also hands the sender a problem the receiver never has to think about with a normal request/response API call: the sender doesn't control when the receiver is down, slow, or returns a 500, yet it still has to get the event there.

The core design question for a webhook sender is what "delivered" means when the receiving endpoint might be unreachable at the exact moment the event fires. There is no way to guarantee exactly-once delivery over an unreliable network — the sender can time out on a request that the receiver actually processed successfully (the ACK was lost, not the request), and retrying that request produces a duplicate. What is achievable is exactly-once processing: the wire delivers at-least-once, and the receiver's own idempotency logic collapses duplicates into a single side effect. This reframing — stop trying to solve delivery guarantees and instead solve processing guarantees — is what makes the problem tractable at all.

That leaves three pieces every reliable webhook system needs, split across both ends:

Idempotency keys (receiver) — every event carries a unique ID; the receiver records IDs it has already processed and no-ops on a repeat, so a retried delivery doesn't double-charge a card or double-send a notification.
Signature verification (receiver) — the payload is signed with a shared secret (typically HMAC-SHA256 over the raw body) so the receiver can reject forged POSTs claiming to be from the sender.
Retry with backoff and a dead-letter path (sender) — failed deliveries (non-2xx, timeout, connection refused) get retried with exponential backoff and jitter, honoring a Retry-After header when the receiver sends one, and after a bounded number of attempts the event is parked in a dead-letter queue for manual replay rather than retried forever.

The sender/receiver contract also has to be fast on the receiver's side: acknowledge with a 2xx as soon as the payload is verified and durably queued, then do the actual slow work (side effects, downstream calls) asynchronously. A receiver that does synchronous heavy lifting before responding risks the sender's own timeout firing and triggering a retry — of an event that's actually already in flight.

Tradeoffs

Reliability choice	Benefit	Cost
No retries (fire-and-forget)	Simplest sender, no dead-letter machinery	Any receiver blip (deploy, transient 500) silently loses events
Retry with exponential backoff + jitter	Absorbs transient receiver downtime without hammering it	Sender must track attempt state per event; receiver may see duplicates during the retry window
Retry + dead-letter queue	Bounds retry cost, gives an operator a place to inspect/replay permanently-failed events	Extra durable storage and a replay workflow someone has to actually own
Idempotency keys on receiver	Makes retries safe — duplicates collapse to one side effect	Receiver needs a dedup store (with its own TTL/eviction policy) keyed on event ID
Synchronous processing before ACK	Simple mental model, no queue	Slow downstream work risks sender timeout → spurious retry of an event already succeeding

The real tradeoff underneath all of these is where the cost of unreliability lands. Fire-and-forget puts the cost on the receiver (silently missing events). Retries move the cost to the sender (state tracking, backoff tuning) and to the receiver (must tolerate duplicates). A dead-letter queue converts "event is lost forever" into "event needs a human to look at it" — worse than automatic recovery, but far better than silent loss.

When to use / when not to

Use webhooks whenever a third party needs to notify you of a real-time event and polling would be wasteful or laggy — payment events, auth-provider user lifecycle events (Clerk, Auth0), CI/CD status callbacks, SaaS integration events.
Treat signature verification as non-negotiable the moment a webhook endpoint is internet-reachable — an unauthenticated webhook URL is an open door for anyone who discovers it to inject fake events.
Build the idempotency/dedup store before the first retry logic ships, not after the first duplicate-charge incident reveals it's missing — retries without idempotency are strictly more dangerous than no retries at all.
Don't reach for webhooks when the caller needs a synchronous answer to act on immediately (use a normal request/response API instead) — webhooks are fundamentally a fire-and-eventually-arrive notification, not a query.
Don't skip the dead-letter path for anything with real consequences (billing events, compliance-relevant state changes) — "retried forever" and "silently dropped after N attempts" are both worse than a bounded retry window with a visible failure queue.

Common pitfall

Treating "the sender got a 2xx" as proof the event was fully processed, and "the sender got a timeout" as proof it wasn't. Neither is true: a 2xx can be returned by a receiver that queued the event but hasn't actually run the side effect yet (a crash before the queued job runs loses the event despite the ACK), and a timeout can happen after the receiver has already durably processed the event but before the response made it back over the network (the sender then retries into an idempotency check that should — but might not, if it wasn't built — safely no-op). The fix is symmetric: the receiver must not ACK until the event is durably persisted (even if the actual side effect runs later, async), and the sender's retry logic must assume duplicates are normal and route every event through the receiver's idempotency check, not just the first attempt.

Engineering Lens

Webhook reliability is a small, self-contained instance of the same distributed-systems truth that shows up everywhere the network sits between "I did the thing" and "you know I did the thing": you cannot make delivery exactly-once, so the design has to stop pretending it can and instead make processing exactly-once, which is a property you actually control. The failure mode to design against isn't "the third party's webhook system is unreliable" — it's the ordinary case, since TCP connections drop, deploys happen mid-request, and load balancers time out idle connections regardless of how well-built either side is. A webhook receiver that has genuinely internalized this treats every incoming event as a possible duplicate by default, the same reflex a payments engineer applies to every charge request — because the two problems are the same problem wearing a different name.