Design a Multi-Region Caching Layer for a Read-Heavy API
Scenario prompt
Design a caching layer for a read-heavy public API (e.g., a product catalog or content-metadata API) serving users across multiple geographic regions. It needs to:
- Cut origin database load and keep read latency low no matter where a request originates
- Stay reasonably consistent when underlying data changes — stale reads should be bounded, not indefinite
- Survive a cache-node failure without triggering a thundering herd on the origin
- Handle a "hot key" (one wildly popular item) without a single cache node buckling
Mihir's attempt
[!todo] Write your own attempt here before reading the model solution below — how many cache layers you'd use, your invalidation strategy, and how you'd protect a hot key.
Model solution
Layer the cache instead of picking one tier. A CDN/edge cache handles cacheable responses close to the user, a shared regional cache cluster (Redis/Memcached) sits in front of the origin DB to absorb cross-instance duplicate reads, and a very-short-TTL in-process cache on each API instance catches the hottest keys before they even hit the network. Each layer exists to solve a different failure mode — geography (CDN), origin load (shared cache), and per-node request rate (local cache) — not as redundant copies of the same thing.
Bound staleness with TTL, but don't rely on TTL alone for freshness. A pure TTL cache means every write is invisible for up to the TTL window, which is fine for some endpoints and unacceptable for others. Pairing a TTL (the staleness ceiling, guaranteeing eventual correctness even if a message is dropped) with event-driven invalidation — a write publishes an invalidation event (via pub/sub or a CDC stream off the DB's write log) that proactively evicts the stale key — gets writes visible fast in the common case while TTL still bounds the worst case.
Protect against thundering herd with request coalescing and jitter. When a hot key expires, a burst of concurrent requests can all miss simultaneously and hammer the origin at once. Request coalescing (a.k.a. single-flight — only the first miss actually queries the origin; concurrent misses for the same key wait on that one in-flight fetch) prevents the duplicate-query stampede. Jittering TTLs (adding random variance instead of a fixed expiry) prevents mass-simultaneous-expiry of many keys that were all set at once (e.g., after a cache flush or deploy). Stale-while-revalidate — serve the expired value immediately while refreshing in the background — trades a little staleness for eliminating the latency spike a synchronous refetch would cause.
Handle a hot key by spreading it, not just caching it harder. Consistent hashing normally sends all requests for a given key to one cache node — great for cache-hit-rate, terrible when that one key is disproportionately popular (a viral product, a trending article). The fix is to detect hot keys and either replicate them across multiple shards (client picks a random replica) or promote them into the local in-process cache tier, so no single shared-cache node has to absorb the full request rate for that key alone.
Gaps to revisit
- Cold-cache stampede after a full regional failover — if a region's cache cluster is lost entirely, traffic rerouted to a healthy region (or a rebuilt cache) hits 100% misses simultaneously. Does the design need pre-warming, or a temporarily relaxed rate limit on origin during recovery?
- Per-endpoint consistency requirements differ — a "product description" endpoint can tolerate seconds of staleness; an "items in stock" endpoint often can't. Does one cache policy fit all endpoints, or does the design need per-endpoint TTL/invalidation tuning?
- Cross-region invalidation lag — if a write happens in one region, how quickly does the invalidation event reach cache nodes in every other region, and what's the user-visible impact of that propagation window?
Principal Engineer Lens
Caching is the pillar-performance pattern that most directly trades off against pillar-cost (bigger/more cache nodes vs. origin load) and consistency (a resilience/correctness concern) simultaneously — a good caching design is really a statement about which staleness the business can tolerate on which endpoint, made explicit rather than left as an accidental side effect of whatever TTL felt reasonable at the time. The request-coalescing and jitter techniques here are also directly transferable to Mihir's network-tooling background — DNS TTL jitter and resolver-level request coalescing solve the exact same stampede problem in a different layer of the stack, which is a good concrete bridge to draw in an architecture review. For a BigTech consumer-platform framing, this is also the shape of problem behind most "read replica vs. cache" design reviews: the answer is almost never "add more DB replicas," it's "cache more precisely."
Reel Script
Setup: A read-heavy public API is getting hammered from users all over the world — how do you keep it fast everywhere without just throwing more database replicas at the problem?
Concept walkthrough: Explain why one cache tier isn't enough — a CDN solves geography, a shared regional cache solves duplicate cross-instance reads, and a local in-process cache solves per-node hot-key pressure — and each is there for a distinct reason, not as a backup for the others. Then explain the TTL-plus-invalidation combo: TTL is the safety net for correctness, invalidation events are what make writes show up fast in practice.
Real example tie-in: Walk through what happens when a hot key's cache entry expires under heavy concurrent load — a stampede of simultaneous origin queries — and how request coalescing (single-flight) and jittered TTLs each independently prevent that pile-up.
Tradeoffs & alternatives: Contrast a simple single-tier TTL cache (easy to build, either too stale or too much origin load depending on the TTL you pick) against the layered design (more moving parts, but lets each endpoint's actual staleness tolerance drive the policy). Note the hot-key problem as the sharp edge that consistent hashing alone doesn't solve.
Principal Engineer takeaway: The real design decision buried in "add a cache" is a staleness-tolerance decision per endpoint, and naming that explicitly — instead of applying one TTL everywhere — is what separates a cache that quietly causes a stale-inventory bug from one that was deliberately designed around the data's actual freshness requirements.