Source: arXiv — 2026-07-08

Summary

PERFOPT-Bench evaluates coding agents on software performance optimization tasks that require profiling code, diagnosing cross-layer bottlenecks, and verifying that a claimed speedup is real rather than a measurement artifact — testing seven different agent stacks across seven long-horizon tasks. It's distinct from an earlier, already-covered paper critiquing whether existing performance benchmarks reliably measure agents at all — this is a new benchmark, not a critique of old ones.

Key Takeaways

• Requires agents to do the full performance-engineering loop: profile the code, diagnose which layer (app, runtime, OS, hardware) actually causes the bottleneck, fix it, then verify the speedup is genuine rather than noise.
• Evaluates 7 distinct agent stacks across 7 long-horizon optimization tasks — a real head-to-head comparison, not a single-model demo.
• Directly addresses a known weak spot in coding-agent evals: agents that report an impressive-looking speedup that turns out to be measurement variance rather than an actual fix.
• Complements rather than duplicates a related already-covered paper questioning the reliability of existing performance benchmarks — that paper critiques measurement methodology broadly; this one is a new, harder benchmark built with that lesson in mind.

Reel Script

Hook Ask a coding agent to make your code faster and it might hand you back a number that looks like a huge win, except sometimes that number is just noise. A new benchmark is built specifically to catch that.

Core Concept Performance optimization is a multi-layer diagnosis problem: a slow function might be slow because of the algorithm, the runtime, memory allocation patterns, or even hardware cache behavior, and picking the wrong layer to fix wastes effort or produces a change that looks good on one run and does nothing on the next. PERFOPT-Bench forces agents through the full real workflow: profile first to find where time is actually going, diagnose which layer is the true bottleneck, apply a fix, then re-measure carefully enough to confirm the speedup holds up and isn't just run-to-run variance — the same discipline a senior performance engineer uses, and the same discipline that's easy to skip when you're an agent that just wants to report a win.

Hands-On The benchmark runs seven different agent stacks against seven long-horizon optimization tasks, comparing not just whether each agent claims a speedup but whether that speedup survives careful re-verification, separating genuine fixes from measurement-artifact wins. That's a meaningful design choice: it's testing the exact failure mode a related paper already flagged as common across coding-agent benchmarks generally, agents and sloppy evals mistaking noise for improvement. Picture it as a seven-by-seven grid where each cell isn't just pass or fail, but genuine speedup, false positive, or no improvement.

Takeaway My verdict: if you're using an agent for performance work, don't trust a reported speedup without re-running it yourself, this benchmark exists precisely because that's an easy trap. Verification discipline matters as much as the fix itself. Follow for more on separating real agent capability from benchmark noise.

Discussion

(No questions yet — ask follow-ups via a Claude Code chat session on this repo; answers get appended here.)