Source: arXiv — 2026-08-04
Summary
PAST-Bench is a new benchmark that tests a claim the agent-memory industry mostly assumes rather than proves: that giving an agent retained memory across sessions actually makes it better at later tasks. Published August 4, 2026, it runs seven base models across four agent frameworks through 26 scenarios and 204 episodes, comparing matched runs with memory switched on versus off, and checks not just whether scores go up but whether the improvement actually follows the intended save-retrieve-update pathway. The finding: gains from retained experience are real, but uneven — some capabilities improve reliably, others show gains that don't trace back to memory working as designed.
Key Takeaways
- The benchmark's core design is a matched on/off comparison — same tasks, same models, memory toggled on or off — isolating whether retained memory itself is what drives any performance gain, rather than something else.
- It spans four capability categories — memory recall, procedural reuse (learned routines), information gathering, and update (revising stored knowledge) — across 26 scenarios and 204 episodes total.
- Tested across seven base models and four agent frameworks, so the results aren't an artifact of one vendor's specific memory implementation.
- Key finding: improvement is real but uneven — agents don't uniformly follow the intended save → retrieve → update loop; some capabilities show gains that are causally traceable to memory, others don't.
- This directly complicates the common pitch from agent-memory products that persistent memory straightforwardly makes agents smarter — PAST-Bench gives a way to check that claim per-capability instead of taking it on faith.
Reel Script
Hook (~18s, 38 words) Every AI memory product promises your agent gets smarter the more it remembers. A new benchmark actually put that claim to the test — and the answer is: sometimes, and not always for the reason you'd assume.
Core Concept (~75s, 165 words) The idea researchers are testing is recursive self-improvement in a narrow, testable sense: an agent does a task, saves something useful from it, retrieves that later, and that changes its behavior for the better next time. The obvious way to test this badly would be to just run an agent with memory on, see the score go up, and declare victory — but a score going up doesn't prove memory caused it. PAST-Bench's fix is a matched comparison: run the exact same agent through the exact same tasks twice, once with memory on and once off, so the only variable that changes is whether it gets to remember. Then it goes a step further and checks the causal path itself — did the gain actually come from the agent saving the right thing, retrieving it correctly, and updating its behavior accordingly, or did the score move for some other reason entirely.
Hands-On (~55s, 130 words) The scale is what makes this credible rather than anecdotal: 26 scenarios, 204 episodes, spread across four capability buckets — memory recall, reusing a learned procedure, gathering information, and updating stored knowledge — tested against seven different base models running inside four different agent frameworks. That breadth is the point: a memory architecture that looks great on one model and one framework could just be overfitting to that combination. The genuinely sharp part of the methodology is the causal check — instead of stopping at "did the later-task score improve," PAST-Bench asks whether that improvement traces back through the intended save-retrieve-update pathway, or whether the agent got lucky some other way.
Takeaway (~22s, 52 words) If you're evaluating an agent-memory product, don't take the vendor's benchmark at face value. Ask whether the specific capability you actually need — recall, procedural reuse, whatever it is — was one of the ones PAST-Bench showed a real, causally-traced gain for, not just an averaged headline number.