Source: arXiv — 2026-07-01
Summary
"Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?" scrutinizes runtime-based coding-agent benchmarks — GSO, SWE-Perf, and SWE-fficiency — which score agents by applying their patches to real repositories and comparing runtime against a reference patch. Re-running these benchmarks in a fresh environment, the authors find leaderboard scores conflate genuine optimization skill with runtime instability, benchmark-specific scoring quirks, and how many tasks a given submission has already "solved" elsewhere.
Key Takeaways
- In cross-machine replay, reference patches only satisfied the original benchmark's own validity rules for 39 of 102 GSO tasks, 11 of 140 SWE-Perf tasks, and 411 of 498 SWE-fficiency tasks.
- SWE-Perf is flagged as especially fragile — many of its reference patches produce close-to-zero actual runtime change, meaning a "pass" can be closer to noise than genuine signal.
- Pairs thematically with — but is a distinct paper and benchmark set from — the recently-covered OpenAI SWE-bench Verified audit, adding a second independent data point on brittle coding-agent evaluation infrastructure.
- Authors: Zhi Chen, Zhensu Sun, Yuling Shi, David Lo, Lingxiao Jiang.