Source: arXiv — 2026-08-06
Summary
This paper combines the AIVAT variance-reduction estimator (originally developed for evaluating poker agents) with continuously-monitored confidence sequences to produce "anytime-valid" stopping rules for agent evaluation — letting evaluators stop testing the moment the evidence is statistically sufficient, rather than committing to a fixed number of trials upfront or invalidating their confidence level by peeking early. The authors report that raw-outcome evaluation needs a median of 74x more trials than AIVAT-corrected evaluation to reach 95% confidence, building on an earlier finding that AIVAT itself already achieves a median 54x variance reduction across 15 LLM agent configurations over 71,439 paired Heads-Up No-Limit Hold'em hands.
Key Takeaways
- The core problem: comparing two agents in a noisy, imperfect-information game (like poker) requires playing enough games that skill outweighs luck — but every game costs money, compute, or expert time, and nobody knows the required number of games in advance.
- Fixed-budget evaluation either wastes money continuing after the result is already clear, or stops too early to reliably tell agents apart; naively stopping early on an ordinary confidence interval invalidates the stated confidence level (a classic "optional stopping" statistical error).
- AV-AIVAT fixes this by combining AIVAT's variance-reduction (which subtracts out predictable, luck-driven outcome variance using a value model trained only on past games, so no game "scores its own correction") with confidence sequences that stay statistically valid no matter when you choose to stop.
- Headline result: raw-outcome evaluation needs a median 74x more trials than AIVAT-corrected evaluation to hit 95% confidence — on top of AIVAT's own previously-reported median 54x variance reduction across 15 LLM agent configurations and 71,439 paired HUNL poker hands.
- Practical implication for AI evals broadly: for any agent-vs-agent evaluation in a noisy or imperfect-information setting, rigorous variance-reduction plus anytime-valid stopping can turn evaluation from a fixed, expensive budget into a cheap, statistically sound "stop as soon as you know" procedure.
Reel Script
Hook: Imagine burning seventy times more compute than you need just to find out which AI agent is actually better. That's not hypothetical — that's the default way most agent evaluations are run today, and a new paper just put a number on the waste.
Core Concept: Say you want to know if your poker-playing AI is actually stronger than the old one. The problem is luck. In any single game, a worse player can still win because the cards fell their way. So you have to play a lot of games until skill clearly outweighs luck in the statistics. But every game costs real money or compute, and here's the catch: you don't know in advance how many games you'll need. If you commit to a fixed number upfront, you either keep paying after the answer's already obvious, or you stop too early and can't tell the agents apart. And if you just peek at your results early and stop whenever it looks good, that's a classic statistics mistake called optional stopping — it silently breaks the confidence level you claim to have. This paper's fix has two parts. First, AIVAT, a technique that strips out the predictable, luck-driven noise from each game's outcome using a value model that only ever learns from past games — so it can't cheat by peeking at the game it's correcting. Second, confidence sequences: a way of computing confidence intervals that stay mathematically valid continuously, so you're allowed to stop the moment the evidence is enough, with the guarantee still intact.
Hands-On: The numbers here are the whole story. AIVAT's variance reduction alone was already shown, across 15 different LLM agent configurations playing 71,439 paired hands of Heads-Up No-Limit Hold'em, to cut the required number of games by a median of 54 times compared to just looking at raw wins and losses. Now stack the anytime-valid stopping rule on top of that, and the paper reports that raw, uncorrected evaluation needs a median of 74 times more trials than the fully AIVAT-corrected, anytime-valid approach to reach 95% confidence. Picture two evaluation runs side by side: one plays out a huge fixed batch of games no matter what, paying for hundreds of games nobody needed; the other watches a live, continuously-valid confidence bar and stops the instant it crosses the line — for a fraction of the cost, with the same statistical guarantee.
Takeaway: If your team is running agent-vs-agent evaluations in any noisy or game-like setting and just counting wins, you're almost certainly overpaying for evaluation by an order of magnitude or more. This is the kind of unglamorous statistics work that should quietly become the new default — go check if your eval pipeline is doing anything this rigorous.