Source: arXiv — 2026-08-11
Summary
Researchers from Mila, Concordia, University of Toronto, Shanghai University, and McMaster University introduce MobileJudgeBench, a benchmark for testing how reliable LLM-as-judge methods actually are when scoring mobile-agent trajectories. Built from 931 human-annotated trajectories across 6 agent benchmarks, 4 agent models, and 68 apps, it evaluates 6 judge methods across 30 method-and-backbone combinations, and finds that a simple screenshot-sampling baseline is competitive with — or beats — several purpose-built judging methods.
Key Takeaways
- Every LLM judge method tested was validated against real human annotations, not just against each other, across 931 trajectories spanning 68 different mobile apps.
- The core finding: a simple baseline judge using sampled screenshots performs on par with or better than more elaborate purpose-built judge architectures — sophistication in the judging method matters less than expected.
- The backbone LLM powering a judge matters more than which judge method it's plugged into, which flips the usual assumption that method design is the lever worth pulling.
- F1 score (not precision alone) is the strongest predictor of whether a judge's rankings and success-rate estimates will match reality; the paper also uses Cohen's Kappa to catch judges that just default to "pass" regardless of the trajectory.
- Anyone using LLM judges to rank mobile agents or as a reinforcement-learning reward signal now has empirical grounds to swap in a cheap, simple judge rather than assuming a complex one is more trustworthy.
Reel Script
Hook: If you're using an AI model to grade another AI model's work, you'd assume a more sophisticated judge gives you a more trustworthy score. A new benchmark just tested that assumption against nearly a thousand human-annotated examples — and the fancy judges didn't win.
Core Concept: When you build a mobile AI agent — something that taps through real apps to complete a task — you eventually need to know if it actually succeeded. Doing that by hand doesn't scale, so teams use another LLM as a "judge" to score the agent's trajectory automatically. The problem is nobody had rigorously tested whether these judges are actually right. MobileJudgeBench fixes that gap: the researchers collected 931 real agent trajectories across 68 different apps, had humans annotate the ground truth for each one, then ran six different judging methods — everything from simple approaches to more elaborate purpose-built systems — and checked every judge's verdict against what a human actually said happened. Think of it like finally fact-checking every referee in a league instead of just trusting the ones with the fanciest whistle.
Hands-On: The mechanism worth sketching is the validation pipeline: real trajectory in, six judge methods score it in parallel, each verdict gets compared against the human-annotated ground truth, and two downstream tests measure whether the judge is actually useful — can it correctly rank agents against each other, and can it serve as a reliable reward signal for reinforcement learning. The surprising result sits right in that comparison: a bare-bones baseline judge that just samples a handful of screenshots from the trajectory scored competitively with, and sometimes beat, the more architecturally complex purpose-built judges. And when the researchers dug into why some judges were more trustworthy than others, it wasn't the judging method that predicted reliability — it was which underlying LLM was powering the judge, measured by F1 score specifically, not precision alone. They even used a statistical correction, Cohen's Kappa, to catch judges that were secretly just rubber-stamping "pass" on everything rather than actually evaluating.
Takeaway: If your evaluation pipeline for AI agents leans on an LLM judge, stop assuming a more elaborate judge architecture buys you more trust — invest that effort in picking a stronger backbone model instead, and validate with F1 against real human labels before you trust any of it. Simple, well-backed judges are the safer default here, not the fallback option.