Source: arXiv — 2026-08-07
Summary
A new paper introduces Agent Memory Distillation (AMD), a training-free framework that transfers a large "teacher" agent's successful problem-solving experience into small "student" agents (4B–8B parameters) purely through structured memory injected at inference time — no fine-tuning involved. The memory is split into three tiers: Workflow memory (task-level strategy), Subtask memory (concrete behavioral examples), and Function memory (tool-calling conventions and pitfalls), distilled from a large teacher model's trajectories and reported to produce measurable gains across three tool-use benchmarks.
Key Takeaways
- No fine-tuning required — the transfer happens entirely through structured memory given to the small model at inference time, not through retraining its weights.
- The three-tier memory split (Workflow / Subtask / Function) separates "what's the overall plan" from "what does a good example look like" from "how do I actually call this tool correctly."
- The teacher model used to generate the source trajectories was GPT-5-mini — itself a relatively small model, showing the technique doesn't require a frontier-scale teacher.
- Reported gains held across three separate tool-use benchmarks, suggesting the effect isn't a one-benchmark fluke.
Reel Script
Hook (17s)
Researchers just showed you can make a small, cheap AI agent noticeably better at using tools — without training it at all. The trick is giving it the right memory, not more parameters.
Core Concept (105s)
Normally, if you want a small, cheap model to perform like a bigger one, you fine-tune it — retrain its weights on examples of good behavior, which costs compute and time. Agent Memory Distillation skips that entirely. Instead, it takes a larger "teacher" model, has it solve a bunch of tasks successfully, and then extracts structured memory from those successful runs — not the raw transcripts, but organized, reusable knowledge — which gets handed to the small "student" model at the moment it's actually working on a task, the same way you'd hand a junior employee a well-organized playbook instead of making them relearn the job from scratch. The playbook has three layers: Workflow memory is the big-picture strategy — "here's roughly how to approach this kind of task." Subtask memory is concrete worked examples — "here's what handling this specific step actually looked like when it went well." Function memory is the fine print — the actual conventions and common mistakes around calling specific tools correctly, the kind of detail that trips up smaller models constantly.
Hands-On (85s)
The genuinely clever part is what "teacher" means here — it's not some enormous frontier model, it's GPT-5-mini, itself a small-to-mid-tier model, generating the source trajectories that get distilled down into memory for even smaller 4B–8B parameter students. That's the mechanism worth sketching: teacher model attempts tasks → successful attempts get parsed into the three memory tiers → memory gets injected into the student's context at inference time → student performs the task with that playbook available. No gradient updates, no retraining loop — just structured context. The paper reports consistent gains across three separate tool-use benchmarks with this setup, which is the evidence that this isn't a one-off benchmark artifact but a real transferable effect.
Takeaway (24s)
If you're running small models in production for cost reasons and hitting a capability ceiling on tool use, structured memory injection is a cheaper lever to pull than fine-tuning — worth prototyping before you reach for a training pipeline. This is exactly the kind of "cheap agent gets smart" result worth testing on your own tool-use tasks.