Source: arXiv — 2026-07-06
Summary
ToolFailBench is a new benchmark aimed at diagnosing how LLM agents fail at tool use rather than just scoring whether the final answer is correct. It targets a blind spot in prior tool-use benchmarks: an agent that never calls a needed tool and one that calls the tool but ignores its output look identical under standard end-task accuracy scoring, even though the failure modes — and the fixes — are completely different.
Key Takeaways
- Spans 1,000 tasks across five domains: finance, medicine, law, cybersecurity, and real estate.
- Explicitly separates "tool-required" tasks (unanswerable by guessing) from "control" tasks (using the same tools but answerable without them), isolating over-reliance versus under-reliance on tools.
- Positioned against prior tool-use benchmarks like ToolSandbox and StableToolBench, which measure end-task success but not the underlying failure mechanism.
- Authored by Harsh Soni.