Source: arXiv — 2026-07-28
Summary
HANDBOOK.md is a new benchmark of 65 agentic tasks that drops an agent into a self-contained simulated company — files, mock email, chat, calendar, issue tracking, and commerce tools, all exposed to the agent over MCP — and instructs it to do routine professional work while obeying an expert-written corporate handbook running 20 to 124 pages. Grading is strict and all-or-nothing: a single prohibited action, or a single omitted required step, fails the whole task even if the actual work product is otherwise correct. The best frontier model tested clears only 36.2% of tasks.
Key Takeaways
- The failure mode this exposes isn't "the agent can't do the job" — it's "the agent does the job but breaks a rule from the handbook along the way," which is a very different and much scarier failure mode for anyone deploying agents into regulated or policy-heavy workflows.
- All-or-nothing grading is intentionally unforgiving because that's how real compliance works: there's no partial credit for "mostly followed the policy."
- The benchmark itself is a data point on MCP's role in agent evaluation — a live company environment (email, Slack, Jira, calendar) is now standard scaffolding to expose over MCP for testing, not just for production deployment.
- The gap between "task completed" and "task completed without violating a single rule" is the number enterprises evaluating agent rollouts should be tracking, not raw task-success rate alone.
Reel Script
Hook (~18s): Every agent demo you've seen shows it finishing the task. Nobody shows what happens when finishing the task means breaking one of your company's own rules on the way there — a new benchmark just measured exactly that, and the best model in the world still fails it most of the time.
Core Concept (~75s): The setup: researchers built a simulated company — complete with files, a mock inbox, mock Slack, a mock calendar, an issue tracker, even mock commerce tools — and wired all of it up as tools the agent can call through MCP, the same protocol behind that GitHub Copilot story. Then they hand the agent a corporate handbook, sometimes over a hundred pages long, and give it one instruction: do this routine task, but follow every rule in the handbook while you do it. Think of it like handing a brand-new employee the full compliance manual on day one and testing whether they can actually get their job done without breaking a rule they just read — except the agent doesn't get to ask a coworker for help, and it doesn't get partial credit. Grading is strict: one banned action, or one skipped required step, and the whole attempt is marked a failure, even if everything else about the work is flawless.
Hands-On (~110s): The number that matters is 36.2% — that's the pass rate of the single best-performing frontier model across the benchmark's 65 tasks, spanning five different enterprise domains. Every other model tested does worse. Rerun that in your head: on well over half of these tasks, even the strongest agent available today either fails the underlying work, or does the work fine and violates a written company rule somewhere in the process. And because the rubric is strict-match, there's no partial credit for "read the handbook, did most of it right" — a single missed disclosure requirement or one prohibited shortcut zeroes out the whole task. That's the gap between "looks impressive in a demo" and "safe to hand a real handbook to."
Takeaway (~25s): If you're evaluating agents for anything touching real company policy — finance, HR, compliance, procurement — raw task-completion rate is the wrong number to trust. Ask what the compliance-violation rate looks like under strict grading, because right now that number is ugly even for the best models. Go check HANDBOOK.md before you take a vendor's demo at face value.