Where Does Agent Reliability Come From? A Cross-Benchmark Decomposition of Verification Loops, Specialist Models, and Scaffolding in a Production Enterprise Agent

Source: arXiv — 2026-07-19

Summary

Researchers studying Leni, a production enterprise agent, decompose exactly where its reliability gains over a frontier base model come from: verification loops (execute, observe, compare, correct), lightweight task-specialized post-trained models, and overall system scaffolding. Evaluated across SpreadsheetBench Verified, BullshitBench v2, and the GAIA validation split, the paper's central and somewhat counterintuitive finding is that scaffolding and routing structure, not the verification checkpoint itself, account for most of the reliability improvement, with the verification step's isolated contribution small but concentrated exactly where it matters most: at the top of the score distribution.

Key Takeaways

• The full system improved over its frontier base model by +11.0 percentage points on SpreadsheetBench Verified (91.25% vs 80.25%, n=400 tasks, p<0.001), +7 to +10 points on BullshitBench v2 (98% vs 91%, n=100), and roughly +15 points on GAIA validation (75.2% pass@1, 83.0% best-of-k, n=165 tasks).
• The three benchmarks were deliberately chosen to stress different failure modes: SpreadsheetBench for silent computation errors, BullshitBench v2 for premise confabulation (the agent inventing a false premise and running with it), and GAIA for cascade errors across long tool chains.
• On GAIA, the gain decomposes cleanly into three separable contributions: planner-executor split (+10pp), cross-provider model routing (+4pp), and per-step verification (+3.6pp), showing structure and routing dwarf verification alone.
• The paper's headline conclusion: most of the reliability improvement comes from scaffolding and specialist-model routing, not from the verification loop itself, whose isolated contribution is small (+1.5pp) but positionally decisive, concentrated at the top of the score distribution where it prevents near-correct answers from being marked as failures.
• Results are grounded in real production evaluation volume: 1,299 runs over 400 SpreadsheetBench tasks, 500 runs over 100 BullshitBench items, and 803 validation runs over 165 GAIA tasks across seven campaigns, plus a 309-run DRACO valid-premise control set.
• Practical implication for teams building enterprise agents: architecture (splitting planning from execution, routing to the right specialist model) is doing more of the reliability work than the verification/self-checking layer that tends to get the most attention.

Reel Script

Hook Everyone assumes the secret to reliable AI agents is having them double-check their own work. A new study on a production enterprise agent found that's mostly wrong, the real gains come from somewhere else entirely.

Core Concept The paper studies Leni, a real production agent, by taking apart exactly which piece of its architecture is responsible for its reliability. There are three candidate explanations: verification loops, where the agent executes a step, observes the result, compares it against expectations, and corrects itself if something's off; specialist models, small models fine-tuned for one narrow sub-task instead of relying on one giant general model for everything; and scaffolding, the overall structural design, like splitting a "planner" role from an "executor" role, or routing different sub-tasks to different model providers. The researchers tested all three against benchmarks built to break agents in different ways: a spreadsheet benchmark for silent math errors, a "bullshit" benchmark for when an agent invents a false premise and runs with it, and GAIA, which punishes long tool chains where one early mistake cascades into a wrong final answer.

Hands-On Here's the decomposition on GAIA, where the breakdown is cleanest: splitting the agent into a separate planner and executor role added about 10 percentage points on its own. Routing different sub-tasks to different specialist models across providers added another 4 points. The verification loop, the part that actually double-checks the agent's work, added only 3.6 points. Across all three benchmarks, the pattern holds: the verification checkpoint's isolated contribution was small, about 1.5 points in the cleanest measurement, but it wasn't useless, its effect concentrated at the very top of the score distribution, catching the near-correct answers that would otherwise get marked wrong. Meanwhile the full system's total gains were substantial: +11 points on SpreadsheetBench, +7 to +10 on BullshitBench, and roughly +15 on GAIA, evaluated across more than 2,600 total runs.

Takeaway If you're building an enterprise agent and betting everything on a self-verification step, this paper says you're optimizing the wrong layer first, architecture and specialist routing are doing most of the work, and verification is a precision tool for the hardest remaining cases, not the main reliability engine. Structure your agent's roles and routing before you invest heavily in self-checking loops.

Discussion

(No questions yet — ask follow-ups via a Claude Code chat session on this repo; answers get appended here.)