Source: arXiv (Microsoft Research) — 2026-08-09
Summary
Standard transformers discard almost everything they computed at each decoding step: only the sampled token feeds back into the next step, while the rich top-layer hidden state that produced it is thrown away. Microsoft Research's "Full-bandwidth transformer" fuses that discarded hidden state back into the next input via a gated linear unit — a technique they call latent feedback — letting unverbalized computation carry forward with a fresh depth budget, while keeping the standard architecture, KV cache, and training objective unchanged.
Key Takeaways
- The problem is a bandwidth mismatch: dense attention gives each token wide horizontal access to everything before it, but the vertical channel carrying information from one decoding step to the next is narrow — just one sampled token's embedding, no matter how much richer computation produced it.
- Latent feedback fuses the previous step's top-layer hidden state with the new token's embedding through a gated linear unit before it re-enters the stack, so information the model "thought" but never wrote down doesn't just vanish.
- Trained at 1B parameters on 400B tokens, the approach improved validation loss along with 5-shot evaluation, math, coding, and instruction-tuned performance versus a standard baseline of the same size.
- Because it preserves the existing KV cache and language-modeling objective, it's a drop-in architectural change rather than a new training paradigm — a low-friction way to close a real inefficiency in every autoregressive transformer in production today.
Reel Script
Hook (17s, ~38 words): Every time a large language model writes a word, it throws away almost all of its own internal reasoning that led to that word. Microsoft Research just published a fix — and it's a small architectural tweak, not a bigger model.
Core Concept (85s, ~195 words): Picture how a transformer generates text: at each step, it processes everything so far, computes a rich internal representation deep inside the network — call it its "current thought" — and then collapses all of that down into one single word to output. That word is the only thing that gets fed back in for the next step. The rich internal thought that produced it? Discarded. Gone. The model has to reconstruct something like it from scratch next step, using only that one word as a clue. That's the bottleneck this paper identifies: transformers have wide "horizontal" bandwidth — each new word can look back at everything previously written — but narrow "vertical" bandwidth between one thinking step and the next. The fix is called latent feedback. Instead of only passing the output word forward, the model also passes its last hidden state — its actual internal thought, not just the word it chose to say — and blends the two together using something called a gated linear unit, which is essentially a smart mixing valve that learns how much of the old thought versus the new word to combine at each step.
Hands-On (70s, ~160 words): What makes this practical rather than just an interesting idea is what it doesn't change. It keeps the standard transformer architecture, the same KV cache mechanism every production system already relies on for fast inference, and the same next-token training objective. It's not a new type of model — it's a rewiring of one feedback path inside an existing one. In testing at 1 billion parameters trained on 400 billion tokens — a meaningful but modest scale, not a frontier-scale run — it beat a standard baseline of the same size on validation loss, 5-shot evaluation benchmarks, math, coding, and instruction-following. That's a broad spread of gains from one architectural change, which is the kind of signal that tends to get scaled up and adopted rather than staying a research curiosity.
Takeaway (25s, ~55 words): This is a "free lunch" style result — same compute budget, same training recipe, better performance — which is exactly the kind of change that tends to quietly show up in the next generation of production models. If you're tracking where model architecture is heading next, latent feedback is worth remembering the name of.