Source: arXiv — 2026-07-14
Summary
This paper replays the actual completed task-level records from three public LLM agent benchmarks — SWE-bench, AppWorld, and tau-bench — to ask a simple but rarely-answered question: how much of the benchmark do you actually need to run before a pairwise "Model A beats Model B" ranking becomes stable? It defines "enough" with three concrete checks (matches the full benchmark's decision, covers required task groups, leaves few comparisons unresolved) and finds the answer varies enormously by benchmark, from 15% of tasks to never-fully-stable.
Key Takeaways
- Defines "enough" rigorously: a partial task budget only counts if it (a) reaches the same pairwise ranking decision the full benchmark would, (b) covers all required task categories, and (c) leaves no more than a target fraction of head-to-head comparisons unresolved.
- Uses real completed task-level data, not simulation, replayed from three public agent benchmarks: SWE-bench (Verified and Lite), AppWorld, and tau-bench.
- Minimum stable task fraction varies wildly: AppWorld first reaches stability at just 15% of its tasks, tau-bench at 25%, SWE-bench Verified needs 90%, and SWE-bench Lite still doesn't reliably reproduce the full benchmark's verdict even at 95% of tasks under the paper's primary coverage rule.
- Implication for anyone running partial benchmark suites to save cost: the same percentage of tasks that's plenty for one benchmark can be nowhere near enough for another — task-set size is an underappreciated source of benchmark noise, separate from task quality or diversity.
Reel Script
Hook (~18s, 40 words): When you read "Model A beats Model B" on an agent benchmark leaderboard, that ranking was often computed from far fewer tasks than you'd assume are needed to trust it. A new replay study puts an actual number on how few is too few.
Core Concept (~65s, 150 words): "Replay analysis" means the researchers took the real, already-completed, task-by-task results from three agent benchmarks — SWE-bench, AppWorld, and tau-bench — and asked: if we'd stopped early, after only some percentage of the tasks, would we have landed on the same verdict about which model wins? They define "enough" with three checks: does the partial run match the full benchmark's final decision, does it cover every required task category, and does it leave only a small share of head-to-head comparisons still undecided. It's the same logic as calling an election from partial precinct counts — sometimes a small sample is plenty to call the race, and sometimes the race is close enough that you need almost every precinct counted before you can trust the call.
Hands-On (~55s, 125 words): Here are the actual thresholds they found. AppWorld stabilizes fastest — just 15% of its tasks is enough to match the full benchmark's ranking decision. Tau-bench needs 25%. SWE-bench Verified needs a much steeper 90% of its tasks before the ranking locks in. And SWE-bench Lite is the outlier: even after running 95% of its tasks, it still doesn't reliably reproduce the full benchmark's verdict under the paper's primary rule. So two benchmarks testing the same broad idea — coding agents — can have completely different noise floors, and there's no way to know which one you're dealing with without checking.
Takeaway (~22s, 50 words): If your team leans on a partial benchmark run to pick a model or gate a release, the benchmark's name matters as much as the percentage completed — some stabilize early, others may never fully stabilize. Before trusting a partial leaderboard number, check which regime that benchmark falls into.