Source: arXiv — 2026-07-09
Summary
This paper compares two ways to spend extra compute for better LLM answers: resampling (rerunning the same model and keeping the best output) versus rerouting (sending the query to a different, usually stronger, model). It shows that under an idealized oracle verifier and unlimited budget, resampling alone recovers much of the quality gain that static routers are built to chase — then proposes a practical "resample-or-reroute" (RoR) online policy that, under a real per-query budget and a realistic imperfect verifier, decides on the fly which option is worth the next unit of spend.
Key Takeaways
- Distinguishes two test-time quality levers that are usually studied separately: resampling (rerun the same model multiple times, keep the best answer) versus rerouting (send the query to a different model, typically larger and pricier).
- Under an idealized oracle verifier — one that always correctly judges which candidate answer is right — and unlimited budget, resampling alone can recover most of the quality gain that a static router is designed to provide, which challenges the assumption that routing beats "just ask again."
- Static routers make one routing decision per query type up front and never revisit it; the paper argues that leaves quality on the table because the best choice can change as budget is spent.
- The proposed resample-or-reroute (RoR) policy makes the choice per query, per budget increment: it estimates the marginal correctness-per-cost of "one more resample from the current model" versus "reroute to a different model," using a realistic, imperfect verifier rather than an oracle, and spends the next unit of budget on whichever option is estimated to buy more correctness per dollar.
Reel Script
Hook (~15s, 38 words): Most "smart routing" systems for LLMs pick a model once per query and stop thinking about it. New research says that leaves free quality on the table — because sometimes the cheapest fix isn't a better model, it's just asking the same one again.
Core Concept (~65s, 145 words): Two different levers get lumped together in test-time compute discussions. Resampling means running the exact same model on the same prompt multiple times and keeping whichever answer checks out best — like asking someone to redo a math problem twice and trusting the version where the steps actually add up. Rerouting means sending the query to a different, usually stronger and more expensive, model instead. Most production routers pick one model per query type up front, based on a guess about difficulty, and never revisit that choice. This paper's striking finding is that under a "perfect judge" — an oracle verifier that can always correctly tell a right answer from a wrong one — plain resampling alone recovers most of the value that smarter routing was supposed to deliver. That's a big deal, because it means routing's advantage may come less from picking a better model and more from just not resampling enough.
Hands-On (~50s, 115 words): The real-world version can't assume a perfect judge or infinite budget, so the paper proposes a live decision loop called resample-or-reroute (RoR). For each query, at each step of spending budget, it estimates two numbers: the marginal correctness-per-cost of spending the next unit on one more resample from the current model, versus spending it on a reroute to a different model — using its imperfect, realistic verifier's best estimate rather than an oracle guarantee. Whichever option is predicted to buy more correctness per dollar right now gets the spend. It's a continuous, per-query, cost-aware decision, not a one-time upfront routing call.
Takeaway (~20s, 48 words): If your product routes LLM traffic to control cost, a router that decides once per query type is probably underperforming one that also asks "should I just try again first?" Worth prototyping a resample-before-reroute fallback before defaulting every hard query to your most expensive model.