Source: arXiv — 2026-08-03
Summary
Self-evolving agents that learn from experience typically store memory as a growing table indexed by every past trajectory, each entry carrying a learned utility score — and as that table grows, feedback gets spread thinner while rewards assigned per-trajectory mistakenly update irrelevant co-retrieved memories too, a failure the authors call the "memory-reward trap." RoMeRL replaces the ever-growing trajectory-indexed table with a fixed-dimensional state per task, factorized by outcome polarity (positive versus negative signal) instead of by individual episode. Across benchmarks, RoMeRL improves task performance while cutting memory size by 84.4% compared to the trajectory-indexed approach.
Key Takeaways
- Trajectory-indexed agent memory grows one entry per past episode, which dilutes feedback across an ever-expanding set of entries as the agent accumulates experience.
- Because rewards are often assigned per-trajectory to a whole batch of co-retrieved memories, irrelevant memories retrieved alongside a good or bad outcome get misleading utility updates — the "memory-reward trap."
- RoMeRL's fix is architectural: instead of one row per trajectory, it keeps a fixed-dimensional utility state per task, split by outcome polarity (roughly, separate positive and negative signal tracks).
- Reported results show RoMeRL both improves task performance and cuts memory size by 84.4% relative to the trajectory-indexed baseline across benchmarks.
Reel Script
Hook (~15-20s, 35-45 words) Self-improving AI agents have a dirty secret — their memory just keeps growing, and it actually makes them worse over time, not better. A new paper shows you can shrink that memory by 84% and get better results at the same time.
Core Concept (~45-90s, 105-200 words) Here's how most self-evolving agents store what they've learned: every time they finish a task, they add a new row to a memory table, one row per past trajectory, each tagged with a "utility" score saying how good that experience was. Sounds reasonable, but it breaks in two ways. First, as that table keeps growing, any new feedback signal gets spread across more and more entries, so each individual memory gets a weaker, noisier update. Second — and this is the sneaky part — rewards usually get handed out per trajectory to a whole batch of memories that were retrieved together, not to the one memory that actually caused the outcome. So if a bad memory happened to get pulled up next to a good one, it gets credit it didn't earn, and vice versa. The authors call this the memory-reward trap. Their fix, RoMeRL, throws out the growing-table idea entirely and replaces it with a fixed-size summary per task, split into positive and negative signal separately.
Hands-On (~45-150s, 105-350 words) Think of the before-state as a spreadsheet that never stops adding rows — one row per episode the agent has ever run, forever. Every new task means more rows, more dilution, and more chances for a reward to land on the wrong memory just because it happened to be retrieved in the same batch. RoMeRL's after-state is a fixed number of columns per task, not a growing number of rows — it keeps a compact utility state and factorizes it by outcome polarity, meaning it keeps a running signal for "what worked" and a separate running signal for "what didn't," instead of one score smeared across thousands of individual trajectory entries. That structural change is what lets feedback stay sharp instead of getting diluted, and it stops irrelevant co-retrieved memories from absorbing credit or blame that wasn't theirs. The reported payoff is concrete: across their benchmarks, RoMeRL both improved task performance and cut memory size by 84.4% compared to the trajectory-indexed baseline. That's not a small pruning tweak — it's a fundamentally more compact way to represent accumulated experience, and it performs better despite storing far less.
Takeaway (~20-30s, 45-70 words) If you're building agents that are supposed to get smarter over time, "just log every trajectory" is the trap — this paper is a solid argument for bounded, outcome-factored memory instead of an ever-growing log. Worth reading if your agent's memory system is starting to feel like a liability rather than an asset.