Architecture/Fundamentals/idempotency-keys

Idempotency Keys

Concept

HTTP gives some methods idempotency for free by definition — GET, PUT (full replace), and DELETE by ID can be safely repeated because repeating them produces the same end state. POST doesn't get this for free: "create a payment" or "place an order" executed twice creates two payments or two orders, which is exactly the failure mode that shows up whenever a client can't tell whether its request actually landed — a timeout, a dropped connection, a load balancer that retried on the client's behalf. The client's only safe move under uncertainty is to retry, and retrying a non-idempotent POST is how double-charges happen.

The idempotency key pattern closes that gap at the application layer. The client generates a unique key (a UUID is the common choice) for each logical operation — not each network attempt — and sends it with the request, typically as an Idempotency-Key header. The server keeps a record per key: the first time it sees a key, it executes the operation and stores the response against that key; every subsequent request carrying the same key returns the stored response without re-executing anything. Three attempts at the same key produce one payment, not three.

The pattern only works if the server does more than just deduplicate by key. It needs to store a fingerprint of the request payload alongside the key, so a client that reuses a key with a different payload gets an explicit conflict instead of silently getting back a cached response for the wrong request. It needs to handle the case where a second identical request arrives while the first is still mid-flight — without an atomic "claim this key" step, two concurrent duplicates can both slip past a naive existence check and both execute. And it needs a retention policy — keys can't be kept forever, so there's an expiry window (Stripe uses 24 hours) after which a repeated key is treated as a brand-new operation.

Tradeoffs

Approach	Duplicate-request safety	Operational cost	Failure mode
No idempotency handling	None — every retry re-executes	Zero	Double-charges, duplicate orders, any retry under network uncertainty corrupts state
Idempotency key, key-only (no payload fingerprint)	Deduplicates identical retries	Low	A key reused with a different payload silently returns the wrong cached response instead of erroring
Idempotency key + payload fingerprint	Deduplicates and detects key/payload mismatch	Moderate — needs a hash comparison per request	None specific to this layer, but still exposed to the in-flight race below if unaddressed
+ atomic claim on first write (lock or conditional insert)	Also safe under concurrent duplicate requests	Higher — needs a datastore with atomic compare-and-set semantics	Without this, two simultaneous duplicates can both pass a read-then-write check and both execute

The underlying tension is where the safety net sits: pure network-level retry logic (client backoff, load balancer retries) assumes the server is idempotent and is silent about whether it actually is, while idempotency keys make the safety explicit but push real implementation cost onto the server — a datastore for the key/response/fingerprint records, a TTL policy, and correct handling of the concurrent-duplicate race. Skipping the payload fingerprint or the atomic claim looks like it works in testing (where retries are rare and sequential) and fails exactly under the production conditions — network flakiness causing rapid concurrent retries — that the pattern exists to handle.

When to use / when not to

Use on any state-changing endpoint the client may retry under uncertainty — payment creation, order placement, resource provisioning, anything where "did that actually go through?" is a real question after a timeout.
Use wherever the client and server are on an unreliable network path — mobile clients, cross-region calls, anything behind a load balancer or proxy layer that might itself retry.
Skip it for naturally idempotent operations — GET, DELETE by ID, PUT with full resource replacement — the HTTP method already provides the guarantee for free.
Skip it for operations with no meaningful "duplicate" concept, like appending a log line where duplicates are harmless or expected.
Don't bolt it onto every POST reflexively — the datastore, fingerprinting, and TTL machinery is real cost, and endpoints where a duplicate side effect is cheap to detect and reverse (rather than prevent) may not need it.

Common pitfall

Storing the idempotency key without a payload fingerprint. It looks correct in the simple case — retry the same request, get the same response — but a client bug that reuses a key across two genuinely different requests (a common mistake when the key is generated per session rather than per operation) silently returns the first request's cached response for the second one. No error, no log line, just quietly wrong data returned to a caller who has no way to know their second request never actually ran.

Principal Engineer Lens

"What happens if this exact request is retried?" is a small question that tends to expose whether a design was actually pressure-tested or just happy-path reviewed — it's a reliable tell in an architecture review because the honest answer usually requires tracing through the datastore layer, not just the API contract. The pattern generalizes past payments: any operation with an external side effect that's expensive or dangerous to duplicate — provisioning infrastructure, sending a downstream notification, submitting an order to an exchange — has the same shape, and the interesting design conversation (how long to retain keys, how to fingerprint a request cheaply, whether the datastore itself can provide the atomic claim or needs an external lock) transfers directly across Fintech, Capital Markets, and general BigTech platform work without modification.

Reel Script

Setup: A mobile client sends "charge this card $50," the request executes successfully on the server, but the response never makes it back before the connection drops. The client, seeing no response, does the only reasonable thing — retries. Now there are two charges for one purchase, and the customer is the one who notices first.

Concept walkthrough: Explain why POST doesn't get idempotency for free the way GET/PUT/DELETE do, then introduce the key pattern: client generates a UUID per logical operation, server stores the key against the first response and returns that stored response on every repeat — no re-execution. Cover the two things that make it actually safe rather than just apparently safe: a payload fingerprint stored alongside the key (catches key reuse with a different payload) and an atomic claim on first write (catches two concurrent duplicates racing each other).

Real example tie-in: Walk through Stripe's implementation — Idempotency-Key header, 24-hour retention window, and the explicit conflict response when a key is reused with a mismatched payload — as the canonical reference implementation most engineers will encounter first.

Tradeoffs & alternatives: Contrast idempotency keys (explicit, server-side, costs a datastore) against relying purely on network-level retry policies that assume idempotency without verifying it. Name the concurrency race explicitly — a naive check-then-write implementation is not actually safe under simultaneous duplicate requests, only under sequential ones.

Principal Engineer takeaway: Any operation with an external side effect that's costly to duplicate needs an answer to "what happens on retry," and the strength of that answer — traced through the actual datastore behavior, not just asserted — is a fast signal of design maturity in a review.

Sources: