Source: arXiv — 2026-07-06 (approximate; arXiv ID 2607.05690 confirms a July 2026 submission)

Summary

A paper by Yusuf Khan and Carlo Lipizzi proposes moving agent memory inside the observe-reason-act loop itself, read and written at every step, rather than treating it as an external store queried once per turn over the network. The authors argue that typical networked vector-database retrieval, at tens to hundreds of milliseconds, inflates end-to-end latency substantially when consulted repeatedly in-loop, and propose an in-process store that answers in roughly 100 microseconds — three orders of magnitude faster — as the threshold needed for memory to function as genuine extended working memory rather than an occasionally-consulted external tool.

Key Takeaways

• The core distinction is architectural placement: memory as an external service the agent queries occasionally, versus memory embedded directly in-process so it can be read and written at every single reasoning step without a network round trip.
• The motivating problem is latency compounding: typical vector-store retrieval over a network takes tens to hundreds of milliseconds per call, and the paper argues this cost becomes prohibitive — up to roughly 83x slower — once an agent tries to consult memory on every step rather than once per turn.
• The proposed fix is an in-process memory store answering in around 100 microseconds, roughly three orders of magnitude faster than typical networked retrieval, which is fast enough to sit inside every reasoning step rather than being reserved for occasional lookups.
• This connects to the "extended mind" thesis from cognitive science — the idea that external tools can functionally become part of a mind's own working memory if they're fast and reliable enough to use unconsciously — the paper's claim is that current RAG-style retrieval is too slow to qualify, but an in-process store might not be.
• The practical implication is a design choice for anyone building long-horizon agents: if memory retrieval is slow enough that it has to be rationed to specific moments, it's functioning as an external tool, not as memory in any meaningful sense — the speed threshold itself changes what the system is architecturally capable of.

Reel Script

Hook: Most "agent memory" systems are so slow that the agent can only afford to check them once in a while — which means it's not really memory, it's more like occasionally googling itself. A new paper argues that's the wrong architecture entirely.

Core Concept: Standard retrieval-augmented generation, or RAG, works by keeping information in a vector database — a separate system the agent queries over a network connection when it needs context. That query typically takes tens to hundreds of milliseconds, which sounds fast until you realize an agent working through a long, multi-step task might want to check its memory at every single step, not just once at the start. Do that consistently and the latency compounds fast — the paper puts the slowdown at up to roughly 83x versus not checking memory in-loop at all. That's the practical reason most agent systems only consult memory sparingly: it's simply too slow to do constantly. The paper's proposal is to stop treating memory as an external service you dial into, and instead build it directly into the process the agent's reasoning loop runs in — the same way your own working memory doesn't require you to "look something up," it's just there, instantly, as part of thinking.

Hands-On: The concrete engineering claim is a speed target: an in-process memory store that answers in roughly 100 microseconds, compared to the tens-to-hundreds of milliseconds typical of networked vector-database retrieval — call it three orders of magnitude faster. That gap is the entire argument. At millisecond latency, checking memory on every reasoning step is expensive enough that you have to ration it, which structurally limits memory to being an occasionally-consulted external tool. At microsecond latency, checking memory becomes cheap enough to do constantly, which is what lets it function as something closer to genuine working memory rather than a lookup service. The paper frames this through the "extended mind" idea from cognitive science — external aids can become functionally part of your own cognition if they're fast and reliable enough that you stop experiencing them as separate lookups at all. Their claim is that current RAG-style retrieval doesn't clear that bar, but an in-process store built for microsecond response times might.

Takeaway: If you're designing a long-horizon agent and your memory system runs over a network call, this paper's argument is worth taking seriously as a real architectural constraint, not just an optimization nice-to-have — the latency of your memory store determines whether it can genuinely function as part of the agent's reasoning process or just as an occasional external reference. Worth prototyping an in-process memory layer if your agent's task genuinely benefits from consulting context at every step rather than once per turn.

Discussion