Source: arXiv — 2026-07-08
Summary
A new paper introduces DeepSWE, a 113-task benchmark for measuring frontier coding agents on original, long-horizon engineering work spanning five languages across 91 active open-source repositories. Unlike benchmarks such as SWE-bench that reuse merged pull requests and their existing tests as ground truth, DeepSWE's tasks are written from scratch and never contributed upstream, so reference solutions can't leak into training data, and each task is graded by a hand-written functional verifier rather than an inherited test suite. An independent LLM-judge audit found DeepSWE's own verifiers disagreed with the judge on only 1.4% of rollouts, versus 32.4% disagreement for SWE-Bench Pro's inherited tests.
Key Takeaways
- The core design flaw DeepSWE targets is pretraining contamination: if a benchmark's tasks come from real merged PRs, the fix and its tests may already exist somewhere in a model's training data, quietly inflating scores.
- DeepSWE avoids this by writing 113 tasks from scratch across 91 active open-source repositories, spanning five languages (TypeScript, Go, Python, JavaScript, and Rust), with solutions that were never contributed upstream and so can't have leaked into any training corpus.
- Grading uses hand-written functional verifiers per task rather than reusing a repo's existing test suite — a verifier checks whether the agent's solution actually works, not whether it happens to match a specific pre-existing patch.
- The paper's headline validation metric is verifier reliability itself: an independent LLM judge auditing rollouts disagreed with DeepSWE's own verifiers only 1.4% of the time, compared to 32.4% disagreement on SWE-Bench Pro's inherited-test grading — a more than 20x gap in measurement noise between the two approaches.
- This matters beyond one benchmark: it's evidence that a meaningful fraction of the gap between "benchmark score" and "actually useful in a real unfamiliar codebase" comes from measurement noise in how existing benchmarks are graded, not just from model capability limits.
Reel Script
Hook: If a coding agent's benchmark score looks suspiciously good, there's a real chance the benchmark's answer key already leaked into the model's training data. A new paper built a benchmark specifically to close that loophole.
Core Concept: Here's the contamination problem in plain terms: popular coding-agent benchmarks like SWE-bench are built from real, merged pull requests on GitHub — take a real bug fix, hide the fix, ask the agent to reproduce it, and grade against the actual PR's own test suite. That sounds rigorous, but it has a hidden flaw: those PRs and their tests are public GitHub history, which is exactly the kind of data large language models are trained on. So a model might not be "solving" the task so much as recalling something adjacent to it from pretraining — like giving a student the exam from a previous semester and being surprised they do well. DeepSWE's fix is structural: every one of its 113 tasks was written from scratch by the researchers and deliberately never merged upstream, so there's no public PR for a model to have memorized. It also swaps out the grading method — instead of reusing a repo's existing tests as the answer key, each task gets a purpose-built verifier that checks whether the agent's solution genuinely works.
Hands-On: The paper's strongest piece of evidence isn't about how well any particular agent scored — it's about whether the grading itself can be trusted. The researchers ran an independent LLM judge over the rollouts and compared its verdicts against each benchmark's own grading. On DeepSWE's hand-written verifiers, the outside judge disagreed with the benchmark's verdict only 1.4% of the time. On SWE-Bench Pro's inherited-test grading, that disagreement rate jumped to 32.4% — meaning roughly one in three graded outcomes on that benchmark are contestable under independent scrutiny. That's not a small gap; it's more than a twentyfold difference in how much you can trust the score you're looking at. The scope is also worth noting: 113 tasks, 91 real active repositories, five languages (TypeScript, Go, Python, JavaScript, Rust) — broad enough to be a genuine cross-language measurement, not a narrow single-ecosystem test.
Takeaway: The practical lesson here isn't about DeepSWE specifically — it's that "our agent scores X% on SWE-bench" should now come with a mental asterisk about how much of that score is real capability versus grading noise or memorized answers. If you're evaluating coding agents for real work, weight benchmarks with contamination-resistant, from-scratch task design far more heavily than ones built from public PR history, because a 32% disagreement rate on the grading itself means the number you're looking at might not mean what it claims to.