Grab: Migrating a Petabyte-Scale Data Lake to Apache Iceberg
Problem + constraints
Grab's data lake grew to petabyte scale on a directory-based Hive/Parquet layout managed through the Hive Metastore. That layout worked at smaller scale, but as tables grew it started fighting the platform on multiple fronts at once: metadata operations against the Metastore got slower as partition counts grew, small-file fragmentation piled up from high-frequency writes (each write creating new files rather than compacting into existing ones), and partition management was a manual, error-prone task that fell on data producers rather than the platform.
The constraint that made this hard wasn't picking a better table format — it was that the lake was already in production, queried continuously by many engines (Spark, Trino, and others) across many teams. A big-bang cutover risked breaking every downstream consumer at once, and any migration path had to keep both the old and new table formats readable by existing query engines throughout the transition.
Solution
Grab evaluated several open table formats and picked Apache Iceberg as the new default, weighing community governance, engine compatibility, and long-term flexibility rather than just raw performance benchmarks — a table format is a decades-long commitment, not a point-in-time optimization.
The rollout was deliberately incremental rather than a lake-wide conversion:
- Highest-value tables migrated first. Rather than converting the entire lake at once, Grab prioritized the tables with the most query traffic and business impact, limiting the blast radius of the migration and front-loading the payoff.
UnifiedSparkCatalog— a catalog abstraction Grab built (and is open-sourcing) to hide table-format differences from users during the transition. Consumers can query a table without needing to know whether it's still Hive/Parquet or already migrated to Iceberg, which decouples the migration schedule from the pace at which every team has to update their own query code.- Z-ordering applied to a high-traffic navigation dataset, co-locating rows by multiple correlated columns on disk so that predicate pushdown can skip far more data via min-max pruning at query time. This alone delivered roughly a 10x improvement in query runtime on that dataset.
What to steal
- Pick a table/storage format for its ecosystem properties, not just its benchmark numbers. Grab's stated selection criteria — community governance, engine compatibility, long-term flexibility — are exactly the properties that determine whether you're still happy with the choice in five years, after the initial performance win has been absorbed into baseline expectations.
- Migrate highest-value assets first, not alphabetically or by convenience. Front-loading the migration on the tables with the most query traffic means the platform team proves value early and the org gets most of the win before the long tail of low-traffic tables is even touched — useful when a migration might get deprioritized partway through.
- Build an abstraction layer to decouple the migration timeline from consumer timelines.
UnifiedSparkCatalogis the same idea as a database read-through proxy during a schema migration: it lets the underlying storage change on the platform team's schedule instead of forcing every consuming team to update code in lockstep. - Physical data layout is a lever independent of the storage format itself. The 10x win came from Z-ordering — a data layout decision — not from Iceberg's metadata format alone. Adopting a modern table format is necessary but not sufficient; the actual query wins often come from a second, deliberate layout pass on top of it.
Engineering Lens
The interesting decision here isn't "Iceberg vs. Hive" — it's the shape of the migration itself. Any sufficiently large, continuously-queried data platform will eventually need a foundational format change, and the naive approaches (stop-the-world conversion, or a fork where old and new tables diverge indefinitely) both fail at scale: one breaks every consumer at once, the other doubles the operational surface forever. The UnifiedSparkCatalog pattern — an abstraction that lets the underlying representation change while the interface consumers see stays constant — is a general-purpose tool for exactly this class of problem, and it shows up anywhere a platform team needs to change something structural without a synchronized fleet-wide cutover: storage engines, serialization formats, even API versions. In an architecture review, when someone proposes "we'll migrate everyone to the new thing over the next two quarters," the sharp follow-up is "what's the abstraction that lets consumers not care which one they're on during those two quarters?" — if there isn't one, the migration timeline is actually gated by your slowest team, not your plan.