Source: arXiv (Muhammad Adnan, Rohan Mahapatra, Prashant J. Nair, Daniel Berger, Pantea Zardoshti, Rodrigo Fonseca, Esha Choukse — Microsoft Research) — 2026-08-06
Summary
Cascade is an LLM serving system that computes one continuously-updated number per request — a "latency budget," meaning the SLO deadline minus the predicted remaining service time — from request features, KV-cache state, and live system load. Unlike prior SLO-aware schedulers that only use deadlines to order the request queue, Cascade uses that same budget number to jointly drive both scheduling and KV-cache tiering: the scheduler prioritizes whichever request has the smallest remaining budget, while the memory manager uses the identical number to decide whether a request's KV state should be prefetched into memory, kept in HBM, or recomputed from scratch. On production traces across three different LLMs, Cascade improved goodput by up to 2.4x and cut SLO violations by 40%, compared to vLLM's default first-come-first-served scheduler.
Key Takeaways
- The core innovation is unifying two decisions that prior systems handled separately: request scheduling (what order to serve requests in) and KV-cache tiering (where to physically keep each request's cached state) — both driven by the same single "latency budget" number.
- Latency budget = SLO deadline minus predicted remaining service time, continuously recomputed from request features, current KV-cache state, and live system load — not a static value set once at request arrival.
- Scheduling policy: prioritize whichever request has the smallest remaining budget, i.e. the request closest to blowing its deadline gets served next — a more dynamic version of standard deadline-ordering.
- Memory policy: the exact same budget number decides whether a request's non-resident KV cache state gets prefetched into memory ahead of need, kept resident in HBM, or simply recomputed from scratch when it's needed again — a tradeoff between memory pressure and recomputation cost that most schedulers don't factor into cache decisions at all.
- Measured results on production traces across three different LLMs: up to 2.4x improvement in goodput (the rate of requests actually completed successfully and on time) and a 40% reduction in SLO violations, compared against vLLM's default first-come-first-served scheduler.
- The significance is architectural: most SLO-aware LLM serving work treats scheduling and memory management as separate subsystems optimized independently; Cascade shows real gains from making them share one decision variable.
Reel Script
Hook (18s)
Two teams inside the same LLM serving stack — the scheduler and the memory manager — are usually making decisions blind to each other. Microsoft's Cascade fixes that with one shared number, and it cuts missed deadlines by 40%.
Core Concept (90s)
When you're serving an LLM in production, you've got a deadline for every request — an SLO, or service-level objective, like "respond within 2 seconds." Two different subsystems have to work together to hit that deadline: the scheduler, which decides which request gets processed next, and the memory manager, which decides where each request's KV cache — its accumulated attention state — physically lives, because moving that state around or recomputing it costs time. The problem with most existing systems is these two subsystems don't talk to each other. The scheduler might know a request is close to its deadline, but the memory manager has no idea, so it might evict that exact request's cached state at the worst possible moment, forcing an expensive recompute right when there's no time to spare. Cascade's fix is elegantly simple in concept: compute one number per request called the latency budget — literally the deadline minus how much service time is predicted to still be needed — and continuously update it as conditions change, using request features, the current KV-cache state, and live system load. Then hand that exact same number to both subsystems. The scheduler uses it to decide who goes next. The memory manager uses it to decide whether to prefetch a request's cache ahead of time, keep it sitting in fast memory, or just let it go and recompute later. One shared signal, two coordinated decisions.
Hands-On (110s)
Walk through what actually happens to a single request under Cascade. It arrives, and the system estimates its remaining service time based on its features and current load, then subtracts that from its SLO deadline to get its latency budget — say the request needs its answer in 2 seconds and predicted work remaining is 1.5 seconds, so its budget is a thin 0.5 seconds. The scheduler sees that thin budget and bumps this request ahead of others with more slack — smallest-remaining-budget-first, essentially triage by urgency rather than simple arrival order. Simultaneously, the memory manager looks at that same 0.5-second number and makes a call: is this request's cache tight enough on time that it needs to be prefetched into fast memory right now, does it already have what it needs resident in HBM, or is there enough slack that recomputing it later from scratch is actually fine and frees up memory for someone more urgent? That's the coordination prior systems were missing. The payoff, measured on real production traces across three different LLMs: up to 2.4x more goodput — meaning requests that actually complete successfully within their deadline — and a 40% drop in SLO violations, both compared against vLLM's default first-come-first-served scheduling, which doesn't reason about deadlines or memory placement at all.
Takeaway (25s)
If you're running LLM inference at any real scale, first-come-first-served scheduling is leaving performance on the table — Cascade's results say the fix isn't a smarter scheduler alone, it's making scheduling and memory placement share the same deadline-aware signal. Worth evaluating against whatever serving stack you're running before assuming your bottleneck is compute.