Spotify: Random Access Parquet — Indexing the Data Lake for Online Point Queries
Problem + constraints
Spotify's online services and AI agents need to look up individual users' data — a full listening history, say — at interactive latency. The problem is scale: petabytes of data fit economically in a key-value store like Bigtable for existing online use cases, but exabytes of Spotify's data actually live in its GCS data lake as Apache Iceberg/Parquet tables. Replicating everything anyone might want to look up into a KV store isn't a real option at that scale — it means paying for a second copy of the data, building and maintaining a pipeline to keep that copy in sync, and accepting that the copy can drift stale relative to the lake.
The query engines that already sit on top of the lake — Trino, BigQuery — don't solve this either, because they're built for analytical throughput, not single-row lookups: even a query that only needs one row pays seconds of job scheduling and query-planning overhead. And even a "just read the file directly" approach is wasteful on its own: a Parquet file's pages can run around 4MB, while the actual record you want might be on the order of 100 bytes — without help, you pay for the whole page to get a sliver of it. A concrete example the team cites: an AI agent asked "what did I listen to last summer" needs one user's slice out of a listening-history dataset spanning billions of users across thousands of large daily files — a full scan to answer that one question isn't viable.
Solution
Spotify built Random Access Parquet (RAP): an external index layered on top of the existing Iceberg/Parquet tables, without touching the underlying immutable Parquet files or storing a second copy of the data.
- The index maps a lookup key — a user ID, for instance — directly to the exact file, row, and byte range where that key's data lives. A query resolves the key through the index first, then issues one precise ranged read against object storage for exactly the bytes needed, instead of a query engine scanning file directories and pulling whole pages.
- An index-builder process generates new, append-only index fragments as new data lands in the Iceberg tables — the index grows alongside the data without ever modifying the Parquet files themselves, preserving the immutability guarantee the rest of the lake's consumers already depend on.
- The optimization is applied selectively, only to the columns that actually serve point-query workloads — other columns keep whatever layout is best for batch analytics, so a table doesn't have to choose one physical layout for every consumer.
- The same physical files keep serving everyone else — ML pipelines, notebooks, experimentation platforms, and batch analytics jobs — because RAP is an index over the existing files, not a specialized copy. The stated design goal is "store once, pay once," rather than accumulating a new serving-optimized copy of the data for every new access pattern anyone invents.
What to steal
- The fix wasn't a faster query engine or a KV-store replica — it was recognizing that query-engine planning overhead is a fixed cost that's simply wrong for a point-lookup workload, and building a narrow index that skips the planning step entirely for that one access pattern while still sharing the underlying storage with everything else.
- Optimizing only the columns that serve the new access pattern, and leaving the rest of the table's layout untouched, avoids a false choice between "optimized for lookups" and "optimized for scans." You don't have to pick one physical layout for a whole table if the workloads genuinely differ column by column.
- Building the index as an append-only, out-of-band artifact that never touches the source files is the generalizable move whenever you want to add a new access pattern to data you don't want to risk mutating or duplicating — the index can be wrong and rebuilt; the source of truth stays untouched throughout.
- "Store once, pay once" is the real cost argument. Every specialized serving copy of a dataset — a KV-store mirror, a search index with its own storage, a cache with its own eviction bugs — is also a second thing to keep consistent, a second storage bill, and a second thing that can silently drift from the source of truth. An index over the data you already have sidesteps all three at once.
Engineering Lens
This is a clean instance of a very general architecture-review question: does this workload actually need a different storage system, or does it need a different access path into the one you already have? The instinctive answer to "we need fast point lookups on lake data" is almost always "replicate it into a KV store" — and sometimes that's the right call, but it's never free: it's a new consistency problem, a new pipeline to operate, and a storage bill that grows with every new consumption pattern someone dreams up later. RAP is what you get when you push back on that instinct and separate what's actually slow — query-engine planning overhead, unindexed full-page reads — from what merely looks slow because "the data lake" gets conflated with "the query engines sitting on top of it." For a Principal-level engineer, the transferable skill is making that separation before reaching for a second copy of the data: the second copy is usually the expensive, hard-to-unwind commitment, while a narrower index over the existing storage is often the cheaper, more reversible fix — and it's the kind of distinction worth raising explicitly in a design review before a team defaults to "let's just replicate it."