DoorDash: A Clusterless Feature Store Serving Layer
Problem + constraints
DoorDash's ML systems need a low-latency serving layer that can read a huge, constantly-refreshed set of precomputed features — the kind of workload that peaks at over 130M HMGET-style lookups per second, resolving to 1.6B+ individual features retrieved per second, all within a 50ms P999 latency budget. Vertical scaling of a single key-value store had already hit its ceiling. The obvious next step, Redis Cluster (or a Kvrocks cluster, which speaks the Redis protocol on top of RocksDB), comes with real overhead: cluster topology management, resharding coordination, and state bookkeeping — none of which DoorDash actually wanted, since their access pattern is closer to "bulk-refresh a huge read-mostly dataset on a schedule" than "handle arbitrary live writes across a mutable keyspace."
Solution
DoorDash built a stateless, clusterless serving layer on Apache Kvrocks (RESP-compatible, RocksDB-backed) instead of adopting Kvrocks' native clustering. Feature data is consolidated from upstream sources via a batch job, written out in Parquet to S3, and then loaded into the serving nodes not through normal key-by-key writes but via SST (Sorted String Table) file sideloading — RocksDB's native bulk-ingest mechanism, which lets a fully-built table file be dropped directly into the storage engine instead of being replayed as millions of individual write operations. Because refresh happens by swapping in new SST files rather than mutating live state, DoorDash could decouple the refresh path from the query path completely: any serving node can be stood up, torn down, or duplicated independently, since it just needs a copy of the current SST files — no cluster membership, no consistent-hashing coordination, no shared mutable state between nodes. That statelessness is what makes the design "infinitely" horizontally scalable in practice: capacity is added by adding identical, independent replicas, not by growing a coordinated cluster.
What to steal
- When your workload is bulk-refresh-then-read-heavy rather than live-mutate, question whether you need a clustered database at all — a stateless fleet of independent nodes fed by bulk-loaded immutable snapshots can be simpler and cheaper to operate than cluster coordination, while still scaling horizontally.
- RocksDB-family engines' bulk-ingest mechanisms (SST sideloading) are a generally underused trick for turning a "write millions of records" refresh into a "swap in a prebuilt file" operation — worth reaching for whenever a batch refresh dominates a serving layer's write path.
- Decoupling the refresh cadence from the query path (batch job -> S3 -> sideload, entirely separate from serving traffic) means a slow or failed refresh degrades staleness, not availability — a resilience property that falls out of the design almost for free.
Principal Engineer Lens
The judgment call worth internalizing here is recognizing when a workload's actual access pattern (bulk-refresh, read-heavy, tolerant of eventual staleness) doesn't need the operational complexity that comes bundled with "the standard clustered solution" for a given data store. It's the same category of tradeoff a Principal engineer has to defend in a review when a simpler, less "distributed-systems-textbook" design outperforms a more conventional clustered one on both cost and operational burden — the win here wasn't a smarter algorithm, it was removing a whole category of coordination problem by exploiting the shape of the workload. Even though the consuming system is ML feature serving, the pattern itself — stateless nodes over bulk-loaded immutable snapshots — is a general caching/storage architecture lesson applicable to any read-heavy, bulk-refreshed serving layer, in Fintech or otherwise.
Reel Script
Setup: DoorDash needed to serve over a billion feature lookups per second at a 50ms P999 latency budget, and vertical scaling of a single key-value store had already run out of headroom.
Concept walkthrough: Explain why Redis/Kvrocks clustering was the obvious next step but came with real coordination overhead the workload didn't actually need, then walk through the clusterless design: stateless serving nodes on Kvrocks/RocksDB, fed by SST file sideloading instead of live writes, so refresh is a file swap rather than millions of individual mutations.
Real example tie-in: Trace the data path end to end — batch job consolidates features, writes Parquet to S3, builds SST files, sideloads them into independent serving replicas — and connect that to why this makes the system horizontally scalable just by adding identical stateless nodes.
Tradeoffs & alternatives: Compare against adopting native Kvrocks/Redis clustering — better suited to workloads with live, arbitrary writes across a mutable keyspace, but overkill (and operationally heavier) for a bulk-refresh, read-heavy access pattern. The clusterless design trades some data freshness (features are as fresh as the last batch refresh) for dramatically simpler operations and near-linear horizontal scaling.
Principal Engineer takeaway: Before defaulting to "the standard clustered solution" for a data store, check whether your actual access pattern (write cadence, mutability, staleness tolerance) even needs the coordination that clustering buys you — removing a whole category of operational complexity is often a bigger win than a smarter algorithm.
Related
Sources: