Developer/AI/AgentMemory-ShortTerm-LongTerm-Summarization/Fundamentals/agent-memory-short-term-long-term-and-summarization

Agent Memory: Short-Term, Long-Term, and Summarization

Concept

An LLM call is stateless by default — everything the model "knows" about a conversation or task is whatever text sits in the context window for that one call. "Memory," for an agent, is purely an engineering layer bolted on top of that stateless call to make information persist and stay retrievable across calls despite the window resetting every time. There is no single right answer for how to do this; production agent frameworks converge on three distinct memory types, each solving a different piece of the problem.

Short-term (working) memory is just the current context window itself — the running message history for the current thread/session, kept intact so the model can see what was just said. LangGraph, for example, manages this as part of agent state persisted via a thread-scoped checkpointer, with a trim_messages utility to cap it at a token budget once it grows too large. It's memory in the weakest sense: it exists only because the window hasn't been cleared yet, and it disappears the moment the thread ends unless something else captures it first.

Long-term memory is external, durable storage the agent explicitly writes to and reads from — a document store, database, or vector index keyed by namespace, persisting facts across sessions rather than just within one. LangGraph's long-term memory support, for instance, is a persistent store with custom namespaces so an agent can recall a specific fact ("this customer's preferred contact method") in a session that has nothing else in common with the session where that fact was learned. Retrieval is on-demand and targeted: the agent (or a retrieval step) looks up only what's relevant to the current step, not the entire history.

Summarization memory sits between the two: instead of keeping the full verbatim history (short-term) or discarding it entirely, older turns are periodically compressed into a running summary that captures "roughly what happened so far." AWS Bedrock AgentCore's Summary Strategy generates condensed, real-time session summaries explicitly so an agent can "recall context without needing to re-process the entire history" — a direct cost/latency argument, not just a context-window-size one.

A realistic agent uses all three at once: short-term memory for the immediate exchange, a summarization pass to keep the token budget from blowing up as the session runs long, and long-term storage for anything that needs to survive past the session entirely.

Tradeoffs

Memory type	What's preserved	Cost/latency profile	Best for
Short-term (in-context)	Full fidelity, verbatim	Free until the window fills, then a hard wall (truncation or failure)	The current exchange, where every detail might matter to the next step
Long-term (external store + retrieval)	Full fidelity, but only for what's explicitly written and later retrieved	Extra write + retrieval round-trip per lookup, but flat cost regardless of session length	Specific facts that must survive across sessions and are looked up on demand, not read wholesale
Summarization (running compressed summary)	Lossy — gist retained, specifics discarded	Cheap to keep in context (small, fixed-ish size) but costs an extra summarization LLM call per compression pass	Long-running sessions where the aggregate picture matters more than any one historical detail

The three aren't competing solutions to the same problem — they're answers to different questions ("what did we just say," "what do we know about X," "what's the gist of everything so far"), and most non-trivial agents need more than one simultaneously.

When to use / when not to

Reach for anything beyond short-term memory as soon as a workflow spans more than one LLM call and a later step needs information from an earlier one that won't reliably fit in context alongside everything else.
Use long-term/retrieval memory for information that's genuinely lookup-shaped — a specific record, preference, or prior decision — not for "everything that ever happened," which just adds retrieval latency without a matching benefit.
Use summarization memory when a session runs long enough that keeping full verbatim history would blow the token budget, and the aggregate picture (counts, trends, "what happened overall") is what later steps actually need — not the individual detail of every earlier step.
Don't reach for summarization when a downstream step needs to reference a specific prior value exactly (an exact number, an exact device ID) — summarization is compression, and compression is lossy by construction; that data belongs in long-term storage with exact retrieval instead.

Common pitfall

Picking a memory type by habit rather than by what a later step actually needs to reconstruct. The failure is easy to walk into: a long-running batch process (say, a 1,000-device rollout) accumulates a running summary of "what happened so far" to keep costs down, and then a later step is asked "what exactly failed on device #742?" — a question the summary was never designed to answer, because summarization memory discards exactly the kind of individual-record detail that question needs. The summary isn't broken; it was built for a different question ("how did the rollout go overall") than the one being asked of it. The fix isn't a better summarization prompt — it's recognizing upfront that individual-record lookups belong in long-term/retrieval memory, and summarization memory should only ever be asked aggregate-shaped questions.

Engineering Lens

Agent memory is the same discipline as tiered storage design in any other system, just relabeled: decide, per piece of data, how far it needs to travel to be useful again, and pick the tier accordingly — L1-cache-equivalent (short-term, free but volatile and space-limited), disk-equivalent (long-term store, durable and precise but costs a round trip), and a lossy cache-equivalent (summarization, cheap and always-available but you can't get the original bits back out). The mistake of asking a running summary for an exact historical value is the same category of mistake as expecting a compressed log file to answer a query that needed the raw, uncompressed record — the compression was a deliberate tradeoff made for a different access pattern, and the fix is always to route the query to the tier that was actually built to answer it.