Source: arXiv — 2026-07-07
Summary
Most agent benchmarks quietly assume a single-language world: the task, the tools, and the output are all in English. PolyWorkBench tests that assumption directly, evaluating long-horizon LLM agents on 67 workplace tasks spanning five domains — commerce, knowledge work, legal analysis, localization, and manufacturing — where inputs, tool outputs, and required outputs mix languages within a single workflow. The headline finding is that state-of-the-art agents suffer significant performance degradation on these multilingual workflows compared to their monolingual performance on equivalent tasks, with the gap compounding as reasoning and tool-use steps stack up over a long task.
Key Takeaways
- The benchmark's core insight is structural: most agent benchmarks implicitly assume reasoning, tool invocation, and output generation all happen in one language — PolyWorkBench instead builds tasks where an agent might receive a document in one language, need to call a tool that returns results in another, and must produce a structured output that respects both.
- Because "correct" for these tasks means more than "right answer" — it also means linguistically consistent and properly formatted — the authors built a hybrid grading framework combining structural grading (does the output match the required schema), executable verification (does the code/action actually work), and LLM-based semantic assessment (is the meaning right across languages).
- The degradation isn't a flat penalty — it compounds. Errors introduced by a multilingual input at one step of a long-horizon task don't stay isolated; they propagate into later reasoning and tool-use steps, making long multilingual workflows disproportionately harder than the sum of their individual multilingual sub-tasks would suggest.
- This directly undercuts a common evaluation shortcut: benchmarking an agent primarily in English and assuming the numbers transfer to deployments serving multilingual users or documents — for global enterprise workflows (legal, localization, manufacturing supply chains), that assumption looks increasingly unsafe based on these results.
Reel Script
Hook: The AI agent benchmark scores you see are almost always measured in English — and a new benchmark shows that when you force the same tasks to mix languages the way real global businesses do, state-of-the-art agents get noticeably worse, not just a little worse.
Core Concept: Long-horizon agent benchmarks usually test whether an agent can plan, call tools, and produce a correct output over many steps — but almost always in a single language throughout. Real work doesn't cooperate: a legal team might get a contract in German, need to query a database that returns results in English, and have to produce a summary in French that a compliance tool checks against a fixed schema. PolyWorkBench builds 67 tasks across five domains — commerce, knowledge work, legal analysis, localization, manufacturing — specifically designed to make agents operate across that kind of language mismatch within a single task, not just translate one document.
Hands-On: Because "did the agent get it right" is harder to judge across languages than in a single one, the paper's grading approach layers three checks: structural grading confirms the output matches the required format/schema, executable verification confirms any code or tool actions actually run correctly, and LLM-based semantic assessment checks that the meaning is preserved and appropriate across the languages involved — a single pass/fail check wasn't enough to catch the kinds of failures multilingual workflows produce. The result that matters most for anyone building production agents: performance degradation compounds over the course of a long-horizon task. A translation slip or a misread multilingual input early in a workflow doesn't just cost points at that one step — it propagates into every downstream reasoning and tool-use decision that depends on it, so the gap between monolingual and multilingual performance widens the longer the task runs.
Takeaway: If you're deploying an agent for anything touching multiple languages — support tickets, cross-border logistics, multinational legal review — don't trust an English-only benchmark score as a proxy for how it'll actually perform. Test it on your real, mixed-language workflow before you trust it with one.