Source: arXiv — 2026-07-31
Summary
LoopsBench, a Microsoft-affiliated benchmark, argues that coding-agent research has over-invested in "harness engineering" (tooling and scaffolding around agents) while under-measuring the iterative reasoning loop agents actually run through on sustained, long-horizon development work. Instead of localized single-shot tasks or end-state pass/fail outcomes, LoopsBench structures 112 tasks as dependency DAGs over independently-testable development units, drawn from authentic repositories spanning 8 programming languages and 9 domains, with more than 5,300 development units and executable tests. The benchmark and code are open-sourced on GitHub as microsoft/Loopsbench.
Key Takeaways
- Core argument: most existing coding-agent benchmarks measure whether a single, localized task got solved, which tells you little about how an agent performs across a long, dependent sequence of development steps — the "loop" of plan, implement, test, revise that real software work actually requires.
- LoopsBench models each of its 112 tasks as a dependency DAG of independently-testable development units (over 5,300 total units), so agent performance can be measured unit-by-unit across a task's full dependency chain, not just at the final outcome.
- Tasks are drawn from authentic repositories across 8 programming languages and 9 domains, aiming for realistic, non-synthetic long-horizon software engineering scenarios rather than curated toy problems.
- Reported headline result: the strongest configuration tested — Opus-4.7 running inside Claude Code with an "outer continuation" strategy — resolves only 25% of tasks, underscoring how much harder sustained, multi-step loops are than single-shot coding benchmarks suggest.
- The benchmark, all tasks, development units, and executable tests are open-sourced (microsoft/Loopsbench on GitHub), making it directly usable by teams evaluating coding agents on long-horizon work rather than isolated bug fixes.
Reel Script
Hook: A top-tier coding agent, running inside one of the best available harnesses, still only finishes one out of every four real long-horizon coding tasks. Not because the tool is bad — because almost nobody is measuring the thing that actually breaks.
Core Concept: Here's the sleight of hand happening across coding-agent marketing right now. Companies pour effort into "harness engineering" — the scaffolding around the model: how it calls tools, how it reads files, how it manages context. And sure, that matters. But LoopsBench's authors point out that real software development isn't one task, it's a loop: you plan a change, implement part of it, run tests, see what broke, revise, and repeat — often across dozens of interdependent pieces of work that build on each other. Most benchmarks test agents on isolated, single-shot problems, or just check whether the final state looks right. That completely misses whether an agent can sustain correct reasoning across a long chain of dependent steps, where an early mistake compounds into later failures. So instead of grading agents on isolated snapshots, LoopsBench breaks each task into a dependency graph — a DAG — of small development units that each have their own tests, chained together the way real feature work actually depends on itself step by step.
Hands-On: The scale here is real: 112 full tasks pulled from authentic repositories, broken down into more than 5,300 independently-testable development units, spanning 8 programming languages and 9 different domains — not a narrow, single-language toy benchmark. And the headline number should give anyone pause: the best-performing setup they tested, Opus-4.7 running inside Claude Code with what they call an "outer continuation" strategy, resolved only 25% of tasks. That's the strongest configuration in the paper, on tasks explicitly designed to test sustained, multi-step, dependency-chained work rather than isolated fixes. The whole benchmark — tasks, all 5,300+ development units, and their executable tests — is open-sourced under microsoft/Loopsbench, so it's not a closed leaderboard claim; anyone can run it themselves.
Takeaway: A 25% success rate from the best configuration on real dependency-chained work is the actual state of coding agents today, not the highlight-reel demos you see online. If you're evaluating agents for long-horizon engineering work, benchmark the loop, not just the harness — go pull LoopsBench and see where your stack actually lands.