Source: arXiv — 2026-07-09
Summary
UniClawBench, from researchers at HKU MMLab and Meituan, is pitched as the first capability-driven (rather than scenario-driven) benchmark for proactive AI agents. Instead of grouping tasks by broad scenarios like "book a trip," it isolates five specific root-cause capabilities — skill usage, exploration, long-context reasoning, multimodal understanding, and cross-platform coordination — across 400 bilingual real-world tasks, graded live inside Docker containers by a closed-loop executor/hidden-supervisor/user-agent triad that never reveals grading criteria to the agent being tested.
Key Takeaways
- Capability-driven, not scenario-driven: existing benchmarks mix multiple skills inside one task category, so a failure on "manage my email" can't tell you whether the agent failed at exploration, memory, or something else — UniClawBench designs each task to isolate one of five specific capabilities.
- 400 bilingual, real-world tasks run inside live Docker containers rather than being scored against static, pre-recorded answer keys.
- Grading is a three-agent closed loop: an executor agent attempts the task, a hidden supervisor agent grades it step by step against criteria the executor never sees, and a separate user agent simulates realistic multi-turn human feedback — preventing the agent under test from gaming a rubric it can't observe.
- Fine-grained, checkpoint-based completion tracking replaces single-turn pass/fail scoring, which is what lets the benchmark actually attribute a failure to a specific one of the five capabilities.
Reel Script
Hook (~15s, 38 words): Most agent benchmarks will tell you a model failed a task. Almost none of them tell you why it failed. A new benchmark called UniClawBench is built specifically to answer that question.
Core Concept (~60s, 140 words): Older agent benchmarks group tasks by scenario — "book a flight," "manage my calendar" — but each of those scenarios secretly requires several different skills at once: finding the right tool, exploring an unfamiliar app, remembering something from twelve turns ago, reading an image correctly, coordinating across platforms. When the agent fails, you genuinely can't tell which of those broke. UniClawBench flips the design: instead of starting from scenarios, it starts from five specific capabilities — skill usage, exploration, long-context reasoning, multimodal understanding, and cross-platform coordination — and builds tasks that isolate one at a time. It's the difference between a doctor telling you "something's wrong" versus running five separate targeted tests and telling you exactly which system is failing.
Hands-On (~50s, 115 words): The grading setup is the real mechanism worth sketching. Tasks run live inside Docker containers, not against pre-written answer keys. Three agents are involved: the executor is the agent actually being tested; a hidden supervisor agent checks its work step by step against criteria the executor is never shown; and a separate user agent plays the human, giving realistic multi-turn feedback as the task unfolds. Because the executor can't see the grading criteria, it can't shortcut or game the rubric — it has to actually demonstrate the capability, not just pattern-match what a grader wants to hear.
Takeaway (~20s, 48 words): If you're building proactive agents, a pass/fail benchmark score won't tell you what to fix next — a capability-isolated one will point straight at whether your gap is exploration, memory, or coordination. Worth checking before you guess where to invest next.