Pinterest: Automated Schema Evolution in a CDC Ingestion Pipeline
Problem + constraints
Pinterest's next-generation change-data-capture (CDC) ingestion framework already solved the hard latency problem: an earlier redesign brought online-to-offline data availability from 24+ hours down to 15 minutes, by streaming DB changes through Kafka and Flink into Iceberg tables instead of running full nightly batch reprocessing. That win exposed a different, structural problem: schema is not just metadata in a CDC pipeline — it's a cross-system contract spanning ingestion, transformation, storage, and historical backfill.
A single table's onboarding is a five-stage pipeline: CDC source configuration, Kafka topic provisioning, Flink and Spark code generation, Iceberg CDC + base table creation, and historical bootstrap — all driven off the same schema definition. Left unmanaged, an upstream schema change can silently break running Flink jobs, block Spark's upsert path, or desync the online (CDC) and offline (base table) representations of the same data. At Pinterest's scale — petabytes of data across thousands of pipelines — handling every schema change as a manual, ad hoc migration doesn't hold up; it becomes the new bottleneck right after the latency problem was solved.
Solution
Pinterest built automated schema-evolution handling directly into the ingestion framework, with a deliberately narrow, safety-first scope:
- Pipeline shape. A custom CDC service (built on Debezium/TiCDC) captures database changes and writes them to Kafka with sub-second latency. Flink jobs process those CDC events near-real-time into CDC Iceberg tables on S3. Every ~15 minutes, Spark jobs read the latest CDC-table changes and apply a
MERGE INTOupsert against the base Iceberg tables that downstream consumers query. - Codegen-driven propagation, reusing the onboarding path. When a supported schema change is detected, an automated workflow regenerates the Flink transformation code, the Spark writer code, and the Iceberg CDC + base table definitions. Critically, it reuses the exact same code-generation logic used to onboard a brand-new table — so an evolved table's generated artifacts are identical to what fresh onboarding would have produced, eliminating drift between "how a table was onboarded" and "how it looks after evolving."
- Scope limited to additive, backward-compatible changes. The automated path only handles changes that preserve backward compatibility (new columns, safe type widening) — deliberately chosen because these changes avoid historical-replay complexity and don't risk breaking existing consumers reading the table.
- Risky changes are routed away from automation, not silently allowed. Type-narrowing changes (e.g.
BIGINT→INT,VARCHAR(200)→VARCHAR(50)) are explicitly rejected by the automated path because they risk data loss; primary-key changes and similarly disruptive edits are routed into a manual or re-onboarding path instead. Automated PR checks catch unsupported changes before they can enter the pipeline at all.
What to steal
- Automate the common, safe case; gate the rare, dangerous one — don't try to automate everything. Pinterest didn't build a general schema-migration engine; it built one that confidently handles additive changes and explicitly refuses to guess on anything that could lose data, pushing that class to a human-reviewed path instead.
- Reuse the onboarding path for evolution instead of building a parallel one. Running the same codegen for "new table" and "changed table" is a small design choice that prevents an entire category of drift bugs where the automated and manual paths quietly diverge over time.
- Make schema a first-class, validated contract, not implicit metadata. Treating schema changes as PR-checkable events (rather than something that just happens to a running table) turns a class of silent production breakage into a caught-before-merge failure.
- Separate the throughput problem from the safety problem, and solve them in that order. Pinterest fixed CDC latency first (24h → 15min) and treated schema safety as the next distinct problem, rather than trying to solve both in one redesign.
Engineering Lens
The interesting decision here isn't the automation itself — it's where Pinterest drew the automation boundary. A team under pressure to reduce manual toil is tempted to automate everything, including the edge cases; Pinterest instead defined "safe" narrowly (additive, backward-compatible only) and routed everything outside that definition to a slower, human-gated path. That's a textbook instance of a pattern that shows up constantly at Principal scope: the goal of automation isn't "never touch this by hand again," it's "make the common case fast and the dangerous case impossible to do by accident." In an architecture review, this is the question worth asking of any proposed automation: what's the failure mode if the automated path guesses wrong, and is that failure mode reversible? Pinterest's answer — reject rather than guess, on anything that could lose data — is the right default for infrastructure that thousands of pipelines depend on, and it generalizes well beyond data pipelines to any system where "mostly correct" automation is worse than a slower, always-correct one.