Architecture/CaseStudies/netflix-kueue-batch-compute-migration

Netflix: Retiring a Homegrown Batch Scheduler for Kueue

Problem + constraints

Netflix built Compute Managed Batch (CMB) in 2018 as a scheduling and queueing layer on top of Titus, its container platform, because nothing in the Kubernetes ecosystem at the time supported what Netflix's internal batch workloads needed: hierarchical tenants, fair sharing across teams, guaranteed capacity reservations, and priority queues. That was the correct call in 2018 — but it meant every improvement, scaling fix, and bug in that scheduling logic has been exclusively Netflix's engineering cost to carry ever since. Eight years later, the broader Kubernetes ecosystem produced Kueue, an open-source, cloud-native job queueing system that now covers much of the same ground CMB was purpose-built for. The constraint on migrating: millions of batch jobs run through CMB across many internal tenants, and the migration had to happen without disrupting any of them or requiring tenants to change how they submit work.

Solution

Kueue is designed to compose with an existing scheduler rather than replace it: it sits above Kubernetes' pod scheduler and only decides whether a job is admitted to run — the queueing, quota, and fairness decision — then hands accepted pods off to the scheduler that's already placing them. That composability let Netflix adopt Kueue without ripping out the Titus scheduling stack underneath it. The migration was primarily a domain-model translation: Netflix's internal tenants became Kueue Cohorts, leaf tenants became a ClusterQueue + LocalQueue pair, and each tenant's existing capacity configuration was converted into Kueue resource flavors and nominal quotas. Titus federation — the layer that already routed jobs across Netflix's compute cells — was extended with a custom Kueue router so jobs land in Kueue-enabled cells. Because the tenant-to-Kueue mapping was configuration rather than code, enrolling a tenant was a single button click for operators in Netflix's internal UI, and just as easily reversible if something went wrong.

The production migration took 4 weeks. Millions of batch jobs moved over. Per Netflix's own account, end users did nothing and noticed nothing — no change to how any tenant submitted or ran batch work.

What to steal

A system built because "nothing in the ecosystem does X" has a shelf life. Revisit that premise periodically — the ecosystem catches up, and every year spent maintaining the homegrown version past that point is a year of exclusive maintenance cost you didn't have to keep paying.
When migrating live infrastructure with many independent tenants, design the cutover as a reversible per-tenant toggle, not a scheduled big-bang switch. A UI button click that's just as easy to click back turns a high-risk global cutover into a series of small, independently reversible steps.
Adopt new foundational infrastructure by composing it on top of what already works (Kueue sits above the existing scheduler) instead of ripping the working layer out — you keep the investment already sunk into the parts that were never the problem.
The hard part of one of these migrations is almost always mapping your existing domain model onto the new tool's primitives (tenants → Cohorts, capacity configs → quotas) — do that translation work explicitly and up front, before touching the mechanics of the swap itself.

Principal Engineer Lens

This reads as a build-vs-maintain story more than a build-vs-buy one. CMB wasn't a mistake — it was the right call given what existed in 2018 — and the Principal-level judgment on display is recognizing the moment a "we had to build this ourselves" system crosses from asset to liability as the surrounding ecosystem matures around it, and being willing to unwind years of homegrown investment rather than defend it on sunk-cost grounds. It's exactly the kind of legacy-migration/modernization tradeoff that shows up at Principal scope regardless of domain — multi-account, multi-team capacity management is core Organizational Complexity territory. It's also a strong reference case for how to sequence a foundational infrastructure swap safely: incremental, reversible, per-tenant enrollment beats a scheduled cutover window whenever the blast radius of getting it wrong is "all batch compute."

Reel Script

Setup: Netflix built its own batch job scheduler back in 2018 because Kubernetes at the time couldn't do multi-tenant fair-share scheduling with priority and capacity reservations. Eight years of exclusively maintaining that homegrown system later, they migrated to the open-source Kueue instead of continuing to own it forever.

Concept walkthrough: Explain Kueue's composable design — it only handles admission (queueing/quota/fairness), then hands off to the existing scheduler, so it layers on top instead of replacing anything. Then walk the migration mechanics: internal tenants map to Cohorts, leaf tenants map to ClusterQueue + LocalQueue, capacity configs map to resource flavors and quotas, and a custom router in Titus federation sends jobs to Kueue-enabled cells.

Real example tie-in: The production migration took 4 weeks and moved millions of batch jobs. Enrollment was a single reversible button click per tenant in Netflix's internal UI — end users noticed nothing changed about how they submitted or ran work.

Tradeoffs & alternatives: Keep maintaining CMB indefinitely — full control, zero migration risk, but permanent exclusive maintenance cost as the ecosystem's tooling keeps improving without you. Adopt Kueue — community-maintained, converges with ecosystem standards, but requires real domain-model translation work up front and a degree of trust in an external project's roadmap and support lifecycle.

Principal Engineer takeaway: Periodically re-ask "do we still need to have built this ourselves?" for any homegrown system that exists because of a gap the ecosystem has since closed — and when you do migrate something foundational, de-risk it through small reversible steps per tenant rather than one big cutover.

Sources: