Source: arXiv — 2026-08-20
Summary
Thinkingbox is a new sandbox and 507-task benchmark for testing AI agents on stateful business workflows — things like retail returns, hospitality bookings, auto-insurance claims, neobank IT tickets, and consulting support, run through isolated, MCP-compatible tool sessions with full execution traces. The paper's core finding is a big gap between "can it succeed once" (pass@1) and "does it succeed every time" (pass^k): the strongest model hit 65.36% pass@1 but only 25.25% pass^20, meaning it reliably nailed just a quarter of tasks across 20 repeated attempts. Many failed runs still terminate cleanly and take valid actions — meaning tool-call success alone is a bad proxy for whether the underlying business task was actually completed correctly.
Key Takeaways
- Thinkingbox-bench: 507 policy-conditioned, stateful business workflow tasks across retail, hospitality, auto insurance, neobank internal IT, and consulting IT/HR support.
- Tasks run in isolated, MCP-compatible tool sessions with complete execution traces, graded by task-specific executable checks against final backend state — not just whether the agent said it succeeded.
- Best model: 65.36% pass@1 (succeeds at least once in one try) but only 25.25% pass^20 (succeeds consistently across 20 repeated attempts) — a roughly 40-point reliability gap.
- Many failed trials terminate cleanly and take valid-looking, state-changing actions, showing that clean termination or successful tool calls are not reliable signals of correct task completion.
- Distinguishes pass@k (odds of at least one success in k tries) from pass^k (odds all k tries succeed) — the metric that actually matters for deploying an agent that has to work every time, not just once.
Reel Script
Hook: An AI agent handled a business task correctly two-thirds of the time when you gave it one shot. Ask it to do that same task twenty times in a row, and it only got it right a quarter of the time. That gap is the whole story.
Core Concept: Most agent benchmarks report "pass@1" — did the agent solve the task once, on a good run. Thinkingbox argues that's the wrong question for real business software, where an agent handling insurance claims or bank IT tickets has to work reliably, not just occasionally. So it measures pass^k instead: the odds that k independent attempts at the same task all succeed. The benchmark itself runs agents through MCP-compatible tool sessions — the same connector standard real agent deployments use — against 507 tasks spanning retail, hospitality, insurance, banking IT, and consulting support, each graded by checking the actual final state of a backend system, not the agent's self-report.
Hands-On: The number that matters: the best-performing model scored 65.36% on pass@1 but crashed to 25.25% on pass^20 — repeat the same task 20 times and it only got a clean sweep about a quarter of the time. Digging into the failures, the paper found many of them looked fine from the outside: the agent terminated cleanly and its tool calls appeared valid, but the end state was wrong — extra side effects, missed steps, or a subtly incorrect final action. That's the paper's real warning: you can't tell an agent is unreliable just by watching whether its tool calls succeed.
Takeaway: If you're deploying an agent for any workflow where "usually works" isn't good enough — billing, insurance, IT provisioning — pass@1 numbers are actively misleading you about production reliability. Before shipping an MCP-connected agent into a stateful backend, run it against the same task repeatedly and check final state, not just successful completion.