Synthesis: HANDBOOK.md Adds a Fourth Axis to the Benchmark-Saturation Story — Policy Compliance, Not Just Task Success
The connection
The existing benchmark-saturation synthesis (2026-07-30) grouped three 2026-07-29 benchmarks — OmniaBench (breadth), GuardianAgentBench (safety), Snorkel's Senior SWE-Bench (difficulty) — as three simultaneous responses to the same problem: agent capability outpacing the evals meant to measure it. HANDBOOK.md, published the very next day (2026-07-30), is a fourth axis that synthesis didn't have yet: policy compliance under all-or-nothing grading, not task success at all.
- HANDBOOK.md drops an agent into a simulated company (files, email, chat, calendar, issue tracking, commerce — all exposed over MCP) with a 20-124 page corporate handbook, and grades strictly: one prohibited action or one omitted required step fails the entire task, even if the work product itself is otherwise correct.
- The best frontier model tested clears only 36.2% — a number lower than GuardianAgentBench's already-alarming 74.8% guardrail ceiling, because HANDBOOK.md isn't measuring "did the agent get exploited by an attacker," it's measuring "did the agent, acting in good faith with no adversary at all, still break one of its own employer's rules while doing routine work."
Why this wasn't visible before
The original synthesis was written 2026-07-30 based on 2026-07-29 items only — HANDBOOK.md landed one day later, in the next day's digest batch, so it was chronologically unavailable when that note was written. It's also tagged differently (benchmarks+mcp rather than benchmarks+security), which would have kept it from surfacing alongside GuardianAgentBench even in a later pass without an explicit cross-check.
What this suggests
- The four benchmarks now form a genuinely complete rubric for evaluating an agent before production deployment: breadth (OmniaBench: does it handle the real diversity of tasks), difficulty (Snorkel: does it handle senior-level, not just junior-level, work), adversarial safety (GuardianAgentBench: does it resist an attacker), and policy compliance (HANDBOOK.md: does it follow the rules even with no attacker present). The fourth axis is arguably the most enterprise-relevant one — most regulated-industry deployments care less about whether an agent can be tricked and more about whether it reliably won't violate a compliance rule during completely normal operation.
- HANDBOOK.md's strict all-or-nothing grading (no partial credit for "mostly followed the policy") is the same design philosophy the regulated-finance hybrid-agent synthesis already identified as the real bar regulated industries hold AI to — "the model said so" isn't acceptable, and now there's a benchmark number (36.2%) quantifying exactly how far frontier agents are from clearing that bar even without any adversary in the loop.
- Worth flagging alongside the coding-agent governance synthesis: HANDBOOK.md's MCP-exposed simulated company (email, Slack, Jira, calendar) is becoming standard evaluation scaffolding, mirroring how GitHub Copilot's newly-GA agent skills + MCP review feature (same week, GitHub Copilot Code Review GA) uses the identical pattern — a
SKILL.md/HANDBOOK.md-style written-policy file plus MCP-mediated live context — for production use, not just evaluation. The eval tooling and the production feature are converging on the same architecture independently.