OmniaBench: Stress-Testing General AI Agents Beyond Narrow Tool-Use Tasks
Source: arXiv — 2026-07-16
Summary
OmniaBench builds a large, hierarchical taxonomy of agent use cases — 90 top-level and 354 second-level domains spanning consumer, business, and enterprise scenarios — mined from app stores, product documentation, and web retrieval. From that taxonomy, it synthesizes executable single- and multi-turn tasks using four distinct generation routes (DAG, DAG-S, Solver, and Program), aiming to test general-purpose agents against the genuine breadth of real-world scenarios rather than the narrow, hand-picked tool-use tasks most existing benchmarks rely on.
Key Takeaways
- The taxonomy behind the benchmark spans 90 level-1 domains and 354 level-2 domains, built by mining app stores, product docs, and web content — an attempt to capture the real breadth of what people actually ask agents to do, not a narrow hand-picked task list.
- Tasks are synthesized via four distinct generation routes (DAG, DAG-S, Solver, Program), each producing executable single- or multi-turn scenarios rather than static Q&A pairs — meaning the benchmark tests agents doing things, not just answering questions.
- Coverage spans consumer (ToC), business (ToB), and enterprise (ToE) use cases in one benchmark, which is unusual — most agent benchmarks focus on a single domain (coding, or customer support, or web browsing) rather than testing generality across all three.
- The explicit goal is measuring "general-purpose" agent capability rather than narrow tool-use skill — a response to the field's benchmarks increasingly clustering around a handful of well-known task types that don't reflect the diversity of actual agent deployments.
Reel Script
Hook (~15s): Most agent benchmarks test one narrow slice of behavior — can it code, can it browse, can it call a weather API. A new benchmark called OmniaBench built a taxonomy of 354 real-world domains instead, and generates actual runnable tasks from it.
Core Concept (~80s): Think about how differently people actually use AI agents day to day — one person wants help drafting a business proposal, another wants an agent to reconcile a spreadsheet, another wants it navigating a shopping app. Most existing benchmarks pick a handful of these scenarios and call it "general agent evaluation," which is a bit like judging a chef's overall skill by watching them make one dish. OmniaBench instead started from the ground up: mining app stores, product documentation, and general web content to build an actual map of what agent use cases look like in the wild — landing on 90 broad domains that break down further into 354 more specific ones, spanning everyday consumer tasks, business workflows, and enterprise scenarios.
Hands-On (~110s): Here's the part worth sketching as a pipeline: taxonomy isn't the end product, it's the input to task generation. The researchers built four separate routes for turning a taxonomy entry into an actual executable task — one based on DAGs (task graphs where steps depend on each other), a variant of that (DAG-S), one built around a solver (working backward from a target outcome to construct the task), and one that generates tasks programmatically. Each route produces something an agent can actually attempt end-to-end, single-turn or multi-turn, rather than a benchmark that's just a list of questions with expected answers. That combination — broad, mined taxonomy feeding into varied, executable task generation — is what lets this benchmark claim to test general capability instead of "can you do this one specific thing well."
Takeaway (~20s): If you're evaluating an agent for a role that isn't "write code" or "browse the web" specifically, benchmarks like this are a better signal of how it'll actually perform than the narrower, more famous ones. Worth checking whether your use case maps onto one of these 354 domains before you trust a model's SWE-bench score to predict how it'll do at your actual job.