In-House LLM Serving at Netflix

Source: Netflix Tech Blog — 2026-07-17

Summary

Netflix's AI Platform team explains why LLM inference lives inside the same unified, JVM-based serving system that already handles the company's recommendation and personalization models, rather than in a separate ML silo or behind a hosted third-party API. The post walks through the concrete engineering choices behind that decision: picking vLLM as the "paved-path" inference engine after starting on TensorRT-LLM, wiring it up behind NVIDIA Triton, exposing both gRPC and OpenAI-compatible HTTP surfaces, and patching Triton's OpenAI-compatible frontend so structured-output requests actually get enforced as guided decoding instead of silently failing. It's a detailed account of production learnings from running embeddings, prefill-only ranking/retrieval, autoregressive decoding, and constrained decoding workloads through one shared system.

Key Takeaways

• Netflix's existing Model Scoring Service (MSS) — a unified, JVM-based system already handling routing, A/B test logic, candidate generation, feature fetching, inference, post-processing, and logging for classic ML — was extended to serve LLMs too, instead of standing up a separate LLM-only stack.
• The team migrated its inference engine from TensorRT-LLM to vLLM by summer 2025 once open-source engines closed the performance gap; vLLM won for its ability to load custom model architectures without heavy compilation steps, its extensibility for custom decoding logic, and better debuggability.
• vLLM runs behind an embedded NVIDIA Triton Inference Server (model loading, batching, GPU scheduling), with a Java control plane on top handling versioning, autoscaling, and multi-region rollout — and both gRPC and OpenAI-compatible HTTP APIs are exposed to callers.
• A real production bug: adopting Triton's OpenAI-compatible frontend as-is silently dropped the response_format field before it reached vLLM, so callers requesting JSON got no guided-decoding enforcement and could silently receive malformed JSON. Netflix git-subtreed and patched the frontend to translate response_format into vLLM's guided-decoding parameters.
• Migrating from vLLM V0 to V1 enabled batch-level constrained decoding, keeping latency flat as batch sizes grew — important since the platform serves a mix of embeddings, prefill-only inference, autoregressive decoding, and custom per-step constraint logic through one shared path.

Reel Script

Hook Netflix could just call an API for every LLM feature it ships. It doesn't. Every recommendation, every ranking call, every generated blurb runs through infrastructure Netflix built and owns — and that choice reveals what "production AI" really costs at hundreds of millions of users.

Core Concept Here's the thing hosted APIs don't give you: control over the whole request lifecycle. When you call an LLM API, you get a black box — you send a prompt, you get text back. But Netflix's real problem was never "how do we call a model," it was "how do we route a request, decide which candidate model or A/B variant should serve it, fetch the right features, run inference, and guarantee the output actually matches a schema — all inside one system, at Netflix's latency and scale requirements." Think of it like the difference between ordering takeout and running your own kitchen. Takeout is fine for one dish. But if you're serving millions of orders a day with dozens of custom recipes, strict dietary constraints, and you need to swap ingredients live without customers noticing — you need the kitchen. Netflix already had that kitchen for its classic recommendation models: a system called the Model Scoring Service. Instead of building a second, separate kitchen just for LLMs, they extended the one they had.

Hands-On Walk the actual pipeline. A request lands on Netflix's unified, JVM-based serving layer — the same layer that's long handled routing and A/B test logic, candidate generation, and feature fetching for classic ML. That part didn't change. What changed is what happens at the inference step. For a while, that step ran on TensorRT-LLM. But by last summer, open-source engines had mostly caught up on performance, and Netflix's workloads had diversified — embeddings, prefill-only inference for ranking and retrieval, full autoregressive generation, custom models with unusual per-step constraints. So they standardized on vLLM as the paved-path engine, chosen because it loads custom architectures without heavy compilation, it's extensible for custom decoding logic, and it's easier to debug. vLLM sits behind an embedded NVIDIA Triton server, which handles model loading, batching, and GPU scheduling, and a Java control plane sits above that, handling versioning, autoscaling, and rollout across regions. Callers reach it over gRPC or an OpenAI-compatible HTTP API. Now here's the part worth remembering: when they first wired up Triton's built-in OpenAI-compatible frontend, they found it silently dropped the response_format field — so a caller asking for guaranteed JSON output got no actual guarantee, no error, just a chance of malformed JSON. They had to fork and patch that frontend so response_format properly translates into vLLM's guided-decoding constraints. Then, migrating from vLLM's V0 to V1 architecture let them enforce those constraints at the batch level, so latency stays flat even as batch sizes grow.

Takeaway Owning your serving stack isn't about avoiding vendors — it's about owning failure modes, like silent schema violations, that a hosted API would hide until they hit production. If you're running LLMs at real scale, read this before assuming "just call the API" is the whole architecture.

Discussion

(No questions yet — ask follow-ups via a Claude Code chat session on this repo; answers get appended here.)