Architecture/CaseStudies/netflix-data-bridge-connector-factory

Netflix Data Bridge: Connector Factory for Cassandra Data Movement

Problem + constraints

Netflix runs a large Cassandra fleet and needs a steady, reliable way to move that data into Apache Iceberg tables for analytics — Netflix's earlier connector for this, Casspactor, handled roughly 1,200 data movements a day, moving on the order of 3 PB out of Cassandra. Casspactor worked, but it wrote everything into one generic, Cassandra-shaped intermediate Iceberg table. Every downstream consumer with a different data model in mind — key-value semantics, time-series semantics, and others — then had to run its own expensive, brittle post-processing pass to reshape that generic output into something usable. As the number of distinct data-model consumers grew, that post-processing step (not the actual data transfer) became both the bottleneck and the main source of pipeline failures. The "move data generically once, then re-specialize per consumer" design didn't scale with the number of specialized shapes people needed.

Solution

Netflix rebuilt this as Data Bridge, a unified control plane for batch data movement, organized around a layered Connector Factory architecture. At the base sits one shared reading engine that reads Cassandra backups directly into standard Spark DataFrames — skipping the old generic intermediate Iceberg table entirely. On top of that shared engine sit thin, data-model-aware connectors (a Key-Value → Iceberg connector, a Time Series connector, and others) that each understand their specific data model's shape and transform the DataFrame straight into final form, with no separate post-processing pass required. Because the hard, shared problem — reliably reading Cassandra backups at scale and standardizing them into a common in-memory format — is centralized in one well-tested engine, adding support for a new data shape becomes a small, focused connector instead of a whole new pipeline. Data Bridge as a control plane also means connector implementations can be swapped or upgraded underneath consumers without those consumers noticing or having to opt in.

What to steal

Collapse "generic staging format + per-consumer post-processing" into "shared engine + specialized thin connectors" once you have more than a couple of distinct consumer shapes — the generic middle format is a recurring tax that scales with the number of consumers, not a one-time cost you pay once.
Push the genuinely hard, shared problem (reliable large-scale reads, standardizing into a common format) down into one well-tested core, and keep the model-specific logic — what this data actually means to a given consumer — thin and swappable at the edges.
A control-plane abstraction that owns "which connector implementation runs" is what makes future connectors additive instead of a rewrite — Netflix notes this is already opening the door to symmetrical work like bulk-loading data back into Cassandra using the same factory pattern.

Principal Engineer Lens

This is a clean example of recognizing that a pipeline's bottleneck had quietly moved — from "can we move the bytes" to "can we reshape the bytes cheaply for N different consumers" — and redesigning around the actual current bottleneck instead of continuing to optimize the part that was already solved. That diagnostic instinct (the constraint you designed for six months ago may not be the constraint you have today) is a recurring Principal-level skill, independent of the specific technology. It generalizes past data platforms: any system with one shared expensive step feeding multiple specialized consumers (a caching layer feeding different read patterns, an event bus feeding different downstream schemas) hits this same shape of problem as the number of consumers grows.

Reel Script

Setup: Imagine a data pipeline that moves petabytes a day without breaking a sweat — and still keeps causing outages downstream. That's the trap Netflix found itself in with Cassandra data movement: the actual data transfer wasn't the problem anymore.

Concept walkthrough: Walk through the old Casspactor model — one generic connector, one generic intermediate Iceberg table shape, and every consumer bolting on its own post-processing to turn that generic shape into what it actually needed. Then introduce the fix: a shared reading engine that goes straight from Cassandra backups to Spark DataFrames, with thin "Connector Factory" connectors on top that are each aware of their own data model (key-value, time-series) and skip post-processing entirely.

Real example tie-in: Trace one dataset through both versions side by side — old path: Cassandra → generic Iceberg table → expensive reshape → consumer. New path: Cassandra → shared DataFrame engine → model-aware connector → consumer, no intermediate table, no separate reshape pass.

Tradeoffs & alternatives: Note the alternative Netflix didn't take — keep optimizing the generic intermediate table format, or ask each consumer team to just write faster post-processing code. Both keep the tax scaling with the number of consumers. The Connector Factory approach costs more upfront design work (a real shared engine, a real plugin interface) in exchange for making the Nth connector cheap instead of the 1st connector cheap.

Principal Engineer takeaway: The skill on display isn't the Spark or Cassandra knowledge — it's noticing that the bottleneck had moved from "move the data" to "reshape the data per consumer," and having the judgment to redesign around the real current constraint instead of polishing the part that already worked. That's the difference between incremental optimization and an architecture that actually holds as the system grows.