Architecture/CaseStudies/spotify-random-access-parquet-data-lake-indexing

Spotify: Random Access Parquet — Indexing the Data Lake for Online Point Queries

Problem + constraints

Spotify's online services and AI agents need to look up individual users' data — a full listening history, say — at interactive latency. The problem is scale: petabytes of data fit economically in a key-value store like Bigtable for existing online use cases, but exabytes of Spotify's data actually live in its GCS data lake as Apache Iceberg/Parquet tables. Replicating everything anyone might want to look up into a KV store isn't a real option at that scale — it means paying for a second copy of the data, building and maintaining a pipeline to keep that copy in sync, and accepting that the copy can drift stale relative to the lake.

The query engines that already sit on top of the lake — Trino, BigQuery — don't solve this either, because they're built for analytical throughput, not single-row lookups: even a query that only needs one row pays seconds of job scheduling and query-planning overhead. And even a "just read the file directly" approach is wasteful on its own: a Parquet file's pages can run around 4MB, while the actual record you want might be on the order of 100 bytes — without help, you pay for the whole page to get a sliver of it. A concrete example the team cites: an AI agent asked "what did I listen to last summer" needs one user's slice out of a listening-history dataset spanning billions of users across thousands of large daily files — a full scan to answer that one question isn't viable.

Solution

Spotify built Random Access Parquet (RAP): an external index layered on top of the existing Iceberg/Parquet tables, without touching the underlying immutable Parquet files or storing a second copy of the data.

The index maps a lookup key — a user ID, for instance — directly to the exact file, row, and byte range where that key's data lives. A query resolves the key through the index first, then issues one precise ranged read against object storage for exactly the bytes needed, instead of a query engine scanning file directories and pulling whole pages.
An index-builder process generates new, append-only index fragments as new data lands in the Iceberg tables — the index grows alongside the data without ever modifying the Parquet files themselves, preserving the immutability guarantee the rest of the lake's consumers already depend on.
The optimization is applied selectively, only to the columns that actually serve point-query workloads — other columns keep whatever layout is best for batch analytics, so a table doesn't have to choose one physical layout for every consumer.
The same physical files keep serving everyone else — ML pipelines, notebooks, experimentation platforms, and batch analytics jobs — because RAP is an index over the existing files, not a specialized copy. The stated design goal is "store once, pay once," rather than accumulating a new serving-optimized copy of the data for every new access pattern anyone invents.

What to steal

The fix wasn't a faster query engine or a KV-store replica — it was recognizing that query-engine planning overhead is a fixed cost that's simply wrong for a point-lookup workload, and building a narrow index that skips the planning step entirely for that one access pattern while still sharing the underlying storage with everything else.
Optimizing only the columns that serve the new access pattern, and leaving the rest of the table's layout untouched, avoids a false choice between "optimized for lookups" and "optimized for scans." You don't have to pick one physical layout for a whole table if the workloads genuinely differ column by column.
Building the index as an append-only, out-of-band artifact that never touches the source files is the generalizable move whenever you want to add a new access pattern to data you don't want to risk mutating or duplicating — the index can be wrong and rebuilt; the source of truth stays untouched throughout.
"Store once, pay once" is the real cost argument. Every specialized serving copy of a dataset — a KV-store mirror, a search index with its own storage, a cache with its own eviction bugs — is also a second thing to keep consistent, a second storage bill, and a second thing that can silently drift from the source of truth. An index over the data you already have sidesteps all three at once.

Principal Engineer Lens

This is a clean instance of a very general architecture-review question: does this workload actually need a different storage system, or does it need a different access path into the one you already have? The instinctive answer to "we need fast point lookups on lake data" is almost always "replicate it into a KV store" — and sometimes that's the right call, but it's never free: it's a new consistency problem, a new pipeline to operate, and a storage bill that grows with every new consumption pattern someone dreams up later. RAP is what you get when you push back on that instinct and separate what's actually slow — query-engine planning overhead, unindexed full-page reads — from what merely looks slow because "the data lake" gets conflated with "the query engines sitting on top of it." For a Principal-level engineer, the transferable skill is making that separation before reaching for a second copy of the data: the second copy is usually the expensive, hard-to-unwind commitment, while a narrower index over the existing storage is often the cheaper, more reversible fix — and it's the kind of distinction worth raising explicitly in a design review before a team defaults to "let's just replicate it."

Reel Script

Setup: Spotify's online services and AI agents need to look up individual users' data — like full listening history — at interactive speed, but that data lives at exabyte scale in a GCS data lake, while only petabyte-scale slices fit economically in a key-value store like Bigtable.

Concept walkthrough: Explain why the existing query engines don't solve this: Trino and BigQuery are built for analytical throughput, so even a single-row lookup pays seconds of job scheduling and query-planning overhead, and reading a Parquet file directly means pulling a whole ~4MB page to get ~100 bytes of actual data. Then introduce RAP: an external index that maps a key straight to its file, row, and byte range, turning a lookup into one precise ranged read against object storage instead of a scan through a query engine.

Real example / case study tie-in: Walk the concrete example — an AI agent asked "what did I listen to last summer" needs one user's slice out of a dataset spanning billions of users across thousands of daily files. Explain the index-builder generating append-only index fragments as new data lands in the Iceberg tables, leaving the underlying Parquet files untouched, and the fact that only the columns serving point queries get this treatment — batch-analytics columns keep their existing layout, and the same files still serve ML pipelines and notebooks exactly as before.

Tradeoffs & alternatives: Contrast RAP against the more obvious answer — replicate the needed slice into Bigtable or another KV store — which is faster to stand up initially but creates an ongoing consistency and cost problem as more services each want their own indexed slice of the same lake. RAP trades a bit of write-path complexity (maintaining index fragments as data lands) for a "store once, pay once" model that scales to new access patterns without new copies.