Source: arXiv (cs.CL) — 2026-08-03
Summary
LiveMem addresses "context turnover" — the fact that in long-running LLM sessions, old tokens eventually get evicted from the active KV cache and the information they held is normally just gone. It adds a fixed-capacity memory state to a pretrained full-attention LLM whose lifetime is decoupled from the active context window, so the main attention path can keep a bounded KV cache while a separate memory state carries information forward through a dedicated write/preserve/read mechanism installed via memory-oriented post-training. On the LongMemEval benchmark, LiveMem still answers correctly even after the supporting evidence has been fully evicted from the active context window, and evidence-distance analysis shows it retaining useful information well beyond that window — posting the leading overall performance among the methods compared.
Key Takeaways
- The problem this targets, precisely: in long-running inference, the KV cache is finite, so old tokens get evicted — and once evicted, the information in them is normally lost for good, even if it becomes relevant again later.
- LiveMem's fix is architectural, not just a longer context window: a fixed-capacity memory state sits alongside the main attention path, with its own lifetime decoupled from the active KV cache window.
- The mechanism is a dedicated write/preserve/read cycle, installed through memory-oriented post-training on top of a pretrained full-attention LLM — it's not prompted behavior, it's trained into the model.
- On LongMemEval, LiveMem answers correctly even when the specific evidence needed for the answer has already been fully evicted from the active context window — direct proof the separate memory state is doing real work, not just the attention window getting lucky.
- Evidence-distance analysis (measuring how far back, past the active window, useful information persists) shows LiveMem retaining information well beyond the window boundary, and it posts the leading overall performance among the methods compared in the paper.
Reel Script
Hook (18s)
Ask an AI a question about something you told it an hour ago, and the tokens holding that answer might already be gone from its working memory. A new architecture called LiveMem answers correctly anyway — even after the evidence is deleted.
Core Concept (85s)
Every LLM conversation runs on something called a KV cache — key-value cache — which is basically the model's short-term working memory for the current conversation. It's what lets the model refer back to something you said five messages ago without re-reading the whole conversation from scratch every single time. The problem is that cache is finite. In a long-running session — hours of back-and-forth, or an agent running for a long task — old tokens eventually have to get pushed out to make room for new ones. That's called context turnover, and normally, once something's evicted from that cache, it's gone. If the answer to your question depended on information that got evicted, the model simply can't retrieve it anymore, no matter how relevant it still is. LiveMem's fix is to stop treating memory as one single pool that everything fights over. It adds a second, separate memory state — fixed in size but with its own independent lifetime, not tied to whatever's currently in the active attention window. Think of it like the difference between your desk — limited space, gets cleared as new papers come in — and a filing cabinet next to it. The model is trained, through a dedicated post-training process, to actively decide what's worth writing into that cabinet, when to preserve it, and when to pull it back out and read it — rather than just letting everything on the desk get thrown away once it's full.
Hands-On (100s)
The proof is in how they tested it. On LongMemEval, a benchmark designed specifically to test whether a model can recall information from far back in a long session, the researchers didn't just check whether LiveMem got the right answer — they checked whether it could get the right answer specifically in cases where the supporting evidence had already been fully pushed out of the active context window. In other words: the tokens that literally contained the information needed to answer the question were gone from the model's normal working memory, and it still answered correctly. That's only possible because the separate memory state had already captured and preserved that information before eviction happened. They also ran what they call evidence-distance analysis — essentially measuring, for correct answers, how far back past the edge of the active window the needed information originally was. The result was that useful information was persisting well beyond the window boundary, not just barely outside it. And on overall benchmark performance, LiveMem came out ahead of the other methods they compared it against.
Takeaway (25s)
For anything building long-running agents or assistants that need to remember earlier context after it's technically been evicted, this is the direction to watch — a trained memory mechanism beats just brute-forcing a bigger context window, because context windows are always finite eventually. Worth reading if your agent's failure mode is "it forgot something from earlier in the session."