Source: arXiv — 2026-08-10
Summary
SWE-Bench ProMax is a new expert-curated benchmark of 170 real-commit refactoring tasks spanning seven programming languages, built partly in response to findings that nearly 60% of unsolved SWE-bench Verified instances have flawed tests. Because refactoring requires coordinated, behavior-preserving changes across many files rather than a single localized bug fix, it's a harder and more realistic test of coding-agent capability — and even the best frontier model-plus-scaffold combination only resolves 41.2% of tasks correctly.
Key Takeaways
- The benchmark was built because existing coding-agent benchmarks like SWE-bench Verified are saturating and have real quality problems — an audit found nearly 60% of its unsolved instances contain flawed test cases, and frontier models can sometimes reproduce gold patches straight from training data rather than actually solving the task.
- Refactoring was chosen specifically because it demands coordinated, behavior-preserving changes across many files at once, which is a fundamentally harder and more realistic test than single-file bug fixes.
- Tasks average 11.4 files and 261.6 lines of code changed — a scale that's much closer to real engineering work than typical single-function benchmark tasks.
- Even the best frontier model and scaffold combination tested resolved only 41.2% of tasks — a hard ceiling that says current coding agents are still far from reliable on real-world-scale refactors, despite topping easier benchmarks.
Reel Script
Hook: Coding agents are topping the leaderboards on the benchmarks everyone uses to compare them. On a new benchmark built from real multi-file refactors, the best one still fails 59% of the time.
Core Concept: There's a known problem with the benchmarks the industry has been using to rank coding agents: they're saturating, and some of them are broken. An audit of SWE-bench Verified — probably the most cited coding-agent benchmark — found that almost 60% of its unsolved instances have flawed tests, meaning the benchmark itself can't reliably tell you whether a failure is the agent's fault or the test's fault. On top of that, some models can reproduce the "gold" fix almost verbatim, which raises the question of whether they're actually reasoning or just recalling something close to it from training data. SWE-Bench ProMax was built to sidestep both problems by testing something structurally different: refactoring. A bug fix is usually localized — change a few lines in one place. A real refactor means changing behavior-preserving code across many files at once, keeping everything consistent, which is a much closer proxy for what senior engineers actually spend their time doing.
Hands-On: The numbers: 170 expert-curated tasks pulled from real commits across seven languages — Python, Java, TypeScript, Go, C, C++, and Rust. The average task touches 11.4 files and changes 261.6 lines of code, which is a genuinely different scale from the single-function tasks most agent benchmarks use. And the headline result: the best frontier model paired with the best agent scaffold tested only resolved 41.2% of these tasks correctly. That's not a benchmark that's saturating — that's a benchmark showing there's real headroom left in coordinated, multi-file code changes.
Takeaway: If your team is leaning on a coding agent for anything beyond isolated bug fixes, this benchmark is the honest picture of where the ceiling actually is right now — treat multi-file refactors as a task that still needs a human reviewing every diff, not something to hand off unsupervised. The 41% number is the real state of the art; don't let a leaderboard on an easier, saturating benchmark convince you otherwise.