Adaptive Adversaries: Why Single-Turn Red-Teaming Understates How Breakable Agents Really Are
Source: arXiv (Jain, Hartmann, Li) — 2026-07-20
Summary
A new benchmark called Adaptive Adversaries tests LLM agent security the way a real attacker would: not with one jailbreak prompt, but with an autonomous LLM attacker that reads the defender's response and pivots, round after round, while the defender itself has no memory of the earlier attempts. Across 21 scenarios, restricting the attack to a single turn found almost nothing wrong (0-1% success), but letting the attacker adapt over 15 rounds pushed success rates up to 14%. The paper also shows that today's top models fail in different, scenario-specific ways rather than sharing one universal weak point.
Key Takeaways
- The core design flaw the benchmark exposes: most agent security evals score a defender's response as a fresh, isolated interaction — so a defender can look safe on every individual turn while still being worn down by an attacker who remembers everything and keeps adjusting its approach.
- Single-turn scoring: 0-1% attack success rate (ASR). The same scenarios, scored after 15 rounds of adaptive multi-turn attack: 5.4-14.0% ASR — the vulnerability was always there, single-turn testing just couldn't see it.
- Using three different frontier LLMs as attackers instead of one surfaced 1.4-2.2x more unique successful attacks, and those attacks had low similarity (cosine 0.02-0.14) to anything in existing benchmarks — meaning a single attacker model, however strong, misses real classes of exploits.
- Claude Opus 4.6 and GPT-5.4 came out statistically tied in aggregate (5.4% ASR each), but that average hides sharply different failure profiles: on one specific scenario, Opus was broken 60% of the time while GPT-5.4 and Gemini stayed under 7% on that same scenario.
- The authors release the full evaluation protocol — scenario plugins, an attacker/defender orchestrator, baseline harnesses, a CLI, plus 945 transcripts and 18,422 recorded attack "battles" — so teams can run adaptive red-teaming against their own agents instead of relying on static prompt-injection lists.
Reel Script
Hook Your AI agent can pass every single-turn security test you throw at it and still be wide open — because the attacker that actually breaks in remembers your last move, and your agent doesn't remember its own.
Core Concept Most agent security testing works like a pop quiz: one adversarial prompt, one response, graded pass or fail, then reset. Adaptive Adversaries throws that model out. Instead, it puts an autonomous attacker LLM in a loop against the defender — the attacker sees exactly how the defender responded last round and adjusts its next attempt accordingly, like a lock-picker who gets to feel which pins gave a little. The defender, meanwhile, is "memoryless" in the sense that every round is scored as if it's the agent's first exposure to that trick — which is exactly how most production agents actually behave, since they don't retain adversarial history across sessions. That mismatch — a learning attacker against a non-learning defender — is the gap the benchmark is built to measure.
Hands-On Here's the number that matters: when you only score the attacker's very first message, these 21 scenarios look almost bulletproof — a 0 to 1 percent success rate. Let that same attacker keep going for 15 rounds, adapting each time based on what it just saw, and the success rate climbs to somewhere between 5.4 and 14 percent. Nothing changed about the agent between those two measurements — only how long the attacker was allowed to keep trying. The researchers also ran three different frontier models as the attacker instead of just one, and found 1.4 to 2.2 times more unique successful attacks than any single attacker model found alone, with those attacks barely overlapping known exploit patterns. And the model comparison is the real gut-check: Claude Opus 4.6 and GPT-5.4 tie on the overall average, both around 5.4 percent — but on one individual scenario, Opus failed 60 percent of the time while GPT-5.4 and Gemini held that same scenario under 7 percent. An aggregate score would have completely hidden that.
Takeaway If your team is red-teaming an agent with one-shot prompts and calling it secure, you're measuring the wrong thing — the real exposure only shows up once you let an attacker adapt over multiple rounds, and it shows up differently for every model, so there's no shortcut to testing your specific stack. Pull the open-sourced harness and run an adaptive, multi-round eval against your own agent before you trust a single-turn score.