Source: arXiv — 2026-07-13

Summary

This paper builds a large-scale empirical taxonomy of how and why CLI coding agents fail across full task trajectories, arguing failure should be studied as a process with distinct stages rather than a single binary outcome — complementing, but distinct from, already-covered work on coding-agent PR-diff taxonomies and benchmark measurement noise.

Key Takeaways

• Reframes "agent failure" as a multi-stage process (misdiagnosis, wrong fix attempt, verification blindness, and similar stages) rather than a single pass/fail label — treating the entire trajectory as the unit of analysis.
• Built from a large-scale empirical sample of real CLI coding-agent trajectories, not a handful of hand-picked anecdotes.
• Distinct from the already-logged coding-agent PR-mutation taxonomy (which studies diff content) and the coding-agent benchmark reliability study (which studies eval measurement noise) — this paper studies the agent's own behavioral failure stages.
• Practically useful for anyone building or debugging a coding-agent harness: knowing which stage failures cluster at (diagnosis vs. verification, for example) tells you where to invest defensive tooling.

Reel Script

Hook When a coding agent fails a task, most benchmarks just log "failed" and move on. A new large-scale study instead maps out exactly where in the process things go wrong, and the answer isn't the same for every failure.

Core Concept Instead of treating agent failure as a single yes-or-no outcome at the end of a task, this study treats a coding agent's attempt as a process with distinct stages: did it misdiagnose the actual bug in the first place, did it correctly diagnose but attempt the wrong fix, or did it fix things correctly but fail to properly verify the fix worked before declaring victory? Those are three completely different problems requiring three completely different fixes to your agent harness. It's the difference between a doctor who misreads an X-ray, one who reads it right but prescribes the wrong medicine, and one who prescribes correctly but never checks if the patient actually got better.

Hands-On Built from a large-scale sample of real CLI coding-agent trajectories rather than hand-picked anecdotes, the study produces a failure taxonomy you can genuinely diagram as a funnel — tasks entering at the top, and at each stage, diagnosis, fix attempt, verification, some fraction dropping out for a distinct reason. That structure is directly actionable: if your own agent setup is failing mostly at the verification stage, the fix is better test-running discipline before the agent claims done; if it's failing at diagnosis, no amount of verification tooling will help, you need better initial-context or search tooling instead.

Takeaway My take: "did the agent succeed" is the wrong question if you're trying to actually improve a coding-agent harness, "where in the process did it fail" is the useful one. This kind of staged failure analysis should be standard practice for any team running agents at scale. Follow for more on debugging agents like real systems, not black boxes.

Discussion

(No questions yet — ask follow-ups via a Claude Code chat session on this repo; answers get appended here.)