Source: Cactus Compute (GitHub) — 2026-08-11
Summary
Cactus Compute released Needle2, a 45-million-parameter agentic language model quantized into a 14MB binary (~28MB RAM per session), built to run tool-calling AI directly on phones, wearables, smart-home devices, and robots — hardware nowhere near a GPU. Despite training on roughly 120x less data than a comparably sized rival, it trades wins with models 5-70x larger on tool-calling benchmarks like BFCL v4, Seal-Tools, and Mobile Actions.
Key Takeaways
- 45M parameters quantized into a 14MB binary; targets ESP32-S3, Raspberry Pi 5, and sub-$200 Android phones.
- Replaces the standard transformer feed-forward layer with a Hadamard MLP — a fixed, weightless orthonormal transform computed in O(n log n) — freeing the parameter budget for other work.
- "Engram" memory looks up key/value pairs from hashed n-gram tables instead of computing full attention, paired with a 256-token sliding window that keeps tool definitions permanently pinned in context.
- Output is constrained by a byte-level grammar compiled directly from each tool's JSON schema, making a malformed tool call structurally impossible rather than just discouraged.
- Trained on ~153B tokens total vs. rival LFM2.5's 19T — yet scores competitively (Mobile Actions 63.7%, BFCL v4 42.6%) against models built on far more data.
Reel Script
Hook: Most people assume a genuinely useful AI agent needs a data-center GPU behind it. A new open-source model just shipped that fits in fourteen megabytes — smaller than a typical podcast episode — and it can still reliably call tools on a fifty-dollar microcontroller.
Core Concept: Here's the trick. Normal transformer models spend a huge chunk of their parameters on something called a feed-forward layer — basically a giant learned lookup table the model consults after every attention step. Needle2 replaces that with a Hadamard MLP: a fixed mathematical transform with zero learned weights, borrowed from signal processing, that reshuffles information almost for free. That frees up the model's actual parameter budget for the part that matters — remembering what it's doing. Instead of full attention over everything it's read, Needle2 uses what its creators call "engram" memory: think of it like an index-card system where past word patterns are hashed into lookup tables instead of recomputed every time, paired with a small sliding window of recent context. And to stop it from ever hallucinating a broken tool call, its output isn't just encouraged to follow the right format — it's mechanically constrained by a grammar compiled straight from the tool's schema, so an invalid tool call isn't just unlikely, it's structurally impossible to generate.
Hands-On: Picture the pipeline as three stations. Input comes in and hits the engram lookup — a hashed table of key-value pairs pulled from n-gram patterns rather than computed fresh, plus a 256-token sliding window with the available tools permanently pinned in memory so the model never forgets what it can call. That feeds into the Hadamard MLP instead of a normal feed-forward block — no learned weights, just a fixed transform doing the reshuffling. Whatever comes out the other end gets filtered through a byte-level grammar compiled from the tool's JSON schema before a single token is finalized — that's the step that makes a malformed function call basically impossible, not just rare. What makes this worth diagramming is the training-data number: Needle2 was trained on roughly 153 billion tokens total. Compare that to LFM2.5, a similarly sized competitor, trained on 19 trillion tokens — about 120 times more data. Despite that gap, Needle2 scores 42.6% on the BFCL v4 tool-calling benchmark and 63.7% on Mobile Actions, trading wins back and forth with models five to seventy times its size, including Google's FunctionGemma and Apple's on-device foundation model. That's the real headline: architecture choices bought back what raw data scale usually buys.
Takeaway: If you're building anything that needs an agent running locally — a wearable, a robot, an appliance with no cloud connection — this is proof the constraint isn't model size anymore, it's smart architecture. Go read the engram and grammar-compiler code before you assume you need a bigger model than you actually do.