DoorDash: A Clusterless Feature Store Serving Layer
Problem + constraints
DoorDash's ML systems need a low-latency serving layer that can read a huge, constantly-refreshed set of precomputed features — the kind of workload that peaks at over 130M HMGET-style lookups per second, resolving to 1.6B+ individual features retrieved per second, all within a 50ms P999 latency budget. Vertical scaling of a single key-value store had already hit its ceiling. The obvious next step, Redis Cluster (or a Kvrocks cluster, which speaks the Redis protocol on top of RocksDB), comes with real overhead: cluster topology management, resharding coordination, and state bookkeeping — none of which DoorDash actually wanted, since their access pattern is closer to "bulk-refresh a huge read-mostly dataset on a schedule" than "handle arbitrary live writes across a mutable keyspace."
Solution
DoorDash built a stateless, clusterless serving layer on Apache Kvrocks (RESP-compatible, RocksDB-backed) instead of adopting Kvrocks' native clustering. Feature data is consolidated from upstream sources via a batch job, written out in Parquet to S3, and then loaded into the serving nodes not through normal key-by-key writes but via SST (Sorted String Table) file sideloading — RocksDB's native bulk-ingest mechanism, which lets a fully-built table file be dropped directly into the storage engine instead of being replayed as millions of individual write operations. Because refresh happens by swapping in new SST files rather than mutating live state, DoorDash could decouple the refresh path from the query path completely: any serving node can be stood up, torn down, or duplicated independently, since it just needs a copy of the current SST files — no cluster membership, no consistent-hashing coordination, no shared mutable state between nodes. That statelessness is what makes the design "infinitely" horizontally scalable in practice: capacity is added by adding identical, independent replicas, not by growing a coordinated cluster.
What to steal
- When your workload is bulk-refresh-then-read-heavy rather than live-mutate, question whether you need a clustered database at all — a stateless fleet of independent nodes fed by bulk-loaded immutable snapshots can be simpler and cheaper to operate than cluster coordination, while still scaling horizontally.
- RocksDB-family engines' bulk-ingest mechanisms (SST sideloading) are a generally underused trick for turning a "write millions of records" refresh into a "swap in a prebuilt file" operation — worth reaching for whenever a batch refresh dominates a serving layer's write path.
- Decoupling the refresh cadence from the query path (batch job -> S3 -> sideload, entirely separate from serving traffic) means a slow or failed refresh degrades staleness, not availability — a resilience property that falls out of the design almost for free.
Engineering Lens
The judgment call worth internalizing here is recognizing when a workload's actual access pattern (bulk-refresh, read-heavy, tolerant of eventual staleness) doesn't need the operational complexity that comes bundled with "the standard clustered solution" for a given data store. It's the same category of tradeoff a Principal engineer has to defend in a review when a simpler, less "distributed-systems-textbook" design outperforms a more conventional clustered one on both cost and operational burden — the win here wasn't a smarter algorithm, it was removing a whole category of coordination problem by exploiting the shape of the workload. Even though the consuming system is ML feature serving, the pattern itself — stateless nodes over bulk-loaded immutable snapshots — is a general caching/storage architecture lesson applicable to any read-heavy, bulk-refreshed serving layer, in Fintech or otherwise.