Source: arXiv — 2026-08-10
Summary
You Lu, Kun Zhang, Bihuan Chen, and Xin Peng show that tool documentation for LLM agents — long treated as a fixed, write-once input — is actually highly dependent on task domain, LLM backbone, and agent paradigm, meaning no single well-written doc block generalizes across setups. Their framework, DocsChisel, analyzes an agent's failed execution traces to find documentation-caused issues, then iteratively adds, removes, and refines specific information fields per tool. Reported results show a 95.89% relative improvement in task success over unoptimized original documentation, and a 75.15% average improvement over existing documentation-optimization baselines, with limited added time and token overhead.
Key Takeaways
- Tool-doc effectiveness is not universal: the same documentation field can help one model/domain/agent-paradigm combination and hurt another, undercutting the assumption that a single well-crafted doc block is enough.
- DocsChisel's core loop is diagnostic rather than generative-from-scratch: it mines an agent's actual failed execution traces to identify which documentation gaps or misleading fields caused the failure.
- Based on that diagnosis, it iteratively edits documentation per tool — adding a missing field, removing an unhelpful one, or refining the wording of an existing field — rather than rewriting everything at once.
- Reported task-success improvement is 95.89% relative to the tool's original, unoptimized documentation.
- Reported average improvement over other existing tool-documentation optimization baselines is 75.15%, achieved with what the authors describe as limited additional optimization time and token overhead.
Reel Script
Hook: Every agent framework tells you to just write clear tool documentation once and move on. New research shows the exact same documentation that helps one model can actively hurt another — and built a system that rewrites your docs automatically by watching your agent fail in real time.
Core Concept: Tool documentation is the instruction manual an LLM agent reads before deciding how to call a function — what parameters it needs, what format to use, what edge cases to watch for. Most agent frameworks treat that manual as a fixed, one-and-done artifact: write it well once, ship it everywhere. This paper's finding cuts against that: the same documentation field can genuinely help one model-domain-agent combination and actively mislead another, because different backbones and different task domains lean on different cues to decide how to call a tool correctly. DocsChisel's fix isn't a smarter one-shot rewrite — it's a diagnostic loop. Instead of guessing what good documentation looks like in the abstract, it watches the actual agent fail on real tasks, figures out whether that failure traces back to a documentation problem, and only then makes a targeted edit to fix exactly that gap.
Hands-On: The loop worth sketching runs in four steps, repeated per tool. First, run the agent on real tasks using its current documentation. Second, collect the failed execution traces and diagnose which failures were actually caused by a documentation issue — a missing parameter format, an unmentioned edge case, a field that led the agent to guess wrong. Third, apply a targeted edit to that specific tool's documentation: add a field that was missing, strip a field that was actively misleading, or refine the wording of a field that was ambiguous. Fourth, re-run and re-test, letting the loop converge toward documentation tuned to that particular domain, backbone, and agent paradigm rather than a generic template. The concrete payoff sits in the two headline numbers: a 95.89% relative task-success improvement over the tool's original, unoptimized documentation, and a 75.15% average improvement over other existing documentation-optimization baselines it was benchmarked against — both achieved while keeping the extra optimization time and token cost limited enough to be practical to actually run.
Takeaway: Tool documentation for agents deserves the same iterative, failure-driven tuning that code and prompts already get, not a write-once-ship-forever treatment — and DocsChisel's failure-trace-mining loop is a solid template for doing that systematically. If you maintain a tool library for agents, start logging which failures are documentation-caused before you hand-edit another doc string.