Architecture/CaseStudies/grab-apache-iceberg-data-lake-migration

Grab: Migrating a Petabyte-Scale Data Lake to Apache Iceberg

Problem + constraints

Grab's data lake grew to petabyte scale on a directory-based Hive/Parquet layout managed through the Hive Metastore. That layout worked at smaller scale, but as tables grew it started fighting the platform on multiple fronts at once: metadata operations against the Metastore got slower as partition counts grew, small-file fragmentation piled up from high-frequency writes (each write creating new files rather than compacting into existing ones), and partition management was a manual, error-prone task that fell on data producers rather than the platform.

The constraint that made this hard wasn't picking a better table format — it was that the lake was already in production, queried continuously by many engines (Spark, Trino, and others) across many teams. A big-bang cutover risked breaking every downstream consumer at once, and any migration path had to keep both the old and new table formats readable by existing query engines throughout the transition.

Solution

Grab evaluated several open table formats and picked Apache Iceberg as the new default, weighing community governance, engine compatibility, and long-term flexibility rather than just raw performance benchmarks — a table format is a decades-long commitment, not a point-in-time optimization.

The rollout was deliberately incremental rather than a lake-wide conversion:

Highest-value tables migrated first. Rather than converting the entire lake at once, Grab prioritized the tables with the most query traffic and business impact, limiting the blast radius of the migration and front-loading the payoff.
UnifiedSparkCatalog — a catalog abstraction Grab built (and is open-sourcing) to hide table-format differences from users during the transition. Consumers can query a table without needing to know whether it's still Hive/Parquet or already migrated to Iceberg, which decouples the migration schedule from the pace at which every team has to update their own query code.
Z-ordering applied to a high-traffic navigation dataset, co-locating rows by multiple correlated columns on disk so that predicate pushdown can skip far more data via min-max pruning at query time. This alone delivered roughly a 10x improvement in query runtime on that dataset.

What to steal

Pick a table/storage format for its ecosystem properties, not just its benchmark numbers. Grab's stated selection criteria — community governance, engine compatibility, long-term flexibility — are exactly the properties that determine whether you're still happy with the choice in five years, after the initial performance win has been absorbed into baseline expectations.
Migrate highest-value assets first, not alphabetically or by convenience. Front-loading the migration on the tables with the most query traffic means the platform team proves value early and the org gets most of the win before the long tail of low-traffic tables is even touched — useful when a migration might get deprioritized partway through.
Build an abstraction layer to decouple the migration timeline from consumer timelines. UnifiedSparkCatalog is the same idea as a database read-through proxy during a schema migration: it lets the underlying storage change on the platform team's schedule instead of forcing every consuming team to update code in lockstep.
Physical data layout is a lever independent of the storage format itself. The 10x win came from Z-ordering — a data layout decision — not from Iceberg's metadata format alone. Adopting a modern table format is necessary but not sufficient; the actual query wins often come from a second, deliberate layout pass on top of it.

Principal Engineer Lens

The interesting decision here isn't "Iceberg vs. Hive" — it's the shape of the migration itself. Any sufficiently large, continuously-queried data platform will eventually need a foundational format change, and the naive approaches (stop-the-world conversion, or a fork where old and new tables diverge indefinitely) both fail at scale: one breaks every consumer at once, the other doubles the operational surface forever. The UnifiedSparkCatalog pattern — an abstraction that lets the underlying representation change while the interface consumers see stays constant — is a general-purpose tool for exactly this class of problem, and it shows up anywhere a platform team needs to change something structural without a synchronized fleet-wide cutover: storage engines, serialization formats, even API versions. In an architecture review, when someone proposes "we'll migrate everyone to the new thing over the next two quarters," the sharp follow-up is "what's the abstraction that lets consumers not care which one they're on during those two quarters?" — if there isn't one, the migration timeline is actually gated by your slowest team, not your plan.

Reel Script

Setup: Grab's data lake had grown to petabyte scale on Hive/Parquet, and the format was starting to fight the platform — slow metadata operations, small-file fragmentation, manual partition management — right as the lake had gotten too large and too widely queried to safely do a big-bang format change.

Concept walkthrough: Walk through why a full-lake cutover was too risky (every engine, every team, querying continuously) and how Grab instead migrated incrementally: highest-value tables first, with UnifiedSparkCatalog hiding the format difference from consumers so query code didn't have to change in lockstep with the migration schedule.

Real example / case study tie-in: Trace the Z-ordering result on the high-traffic navigation dataset — co-locating correlated columns on disk so query engines can skip far more data via min-max pruning, landing a roughly 10x query-runtime improvement. Make the point explicit: this gain came from a data-layout decision layered on top of Iceberg, not from switching table formats alone.

Tradeoffs & alternatives: Contrast a stop-the-world lake conversion (simpler to reason about, but breaks every consumer simultaneously and is nearly impossible to roll back) against Grab's incremental, abstraction-fronted migration (slower to fully complete, but each step is independently safe and reversible). Note the real cost of the safer path: building and maintaining UnifiedSparkCatalog is itself extra engineering investment that only pays off because the migration is large enough to need it.