Source: arXiv — 2026-07-17
Summary
A new paper from Tsinghua University researchers (Wen, Zhang, Chen, Lyu) shows that even frontier language models fail at exact-copy tasks — reproducing a span of text verbatim — well within their context windows, and traces the failure to a specific architectural cause: positional encodings that bias models toward matching local context rather than tracking true token position. Their fix, 2D-RoPE, arranges tokens on a 2D grid instead of a 1D sequence, turning copying into a fixed-offset retrieval problem; shallow transformers trained with it copy near-perfectly at input lengths hundreds of times longer than anything seen in training, and the advantage holds at pretraining scale up to 1.4B parameters on the DCLM dataset.
Key Takeaways
- Exact copying — reproducing a passage verbatim — sounds trivial, but frontier LLMs get it wrong inside their own context window; the paper attributes this to positional encoding's inductive bias toward a "local context matching" shortcut instead of genuine position tracking.
- Standard 1D positional schemes (like RoPE) encode a token's position as a single number along a line, which works fine for generation but gives the model no clean way to say "fetch the token N positions back" without confusing similar-looking local contexts.
- 2D-RoPE assigns each token a row ID and a column ID, arranging text on a grid; under this scheme, copying a span reduces to reading off a fixed column offset — a much easier pattern to learn than searching a 1D sequence for the right anchor.
- Reported results: shallow transformers with 2D-RoPE achieve near-perfect copy accuracy at input lengths hundreds of times longer than their training length, dramatically outperforming standard 1D positional encoding baselines.
- The advantage isn't just a toy-scale result — the paper reports it holds through large-scale pretraining on the DCLM dataset at model sizes up to 1.4B parameters, suggesting this is a fixable architectural property, not an inherent LLM limitation.
Reel Script
Hook (~18s, 40 words) Ask a frontier model to copy a paragraph word-for-word from earlier in the same conversation, and it will quietly drop a word or reorder a phrase — well within its context window. That's not a training gap. It's a structural blind spot in how transformers track position.
Core Concept (~65s, 145 words) Here's the mechanism. Every transformer needs to know where each token sits in a sequence, so it uses something called positional encoding — think of it as stamping each word with a coordinate. The standard method, RoPE, stamps tokens with a single number along a line, like house numbers on a long street. That works great for predicting the next word, but it's a bad fit for exact copying: instead of learning "go find the token that's exactly 500 positions back," the model learns a shortcut — "find something that looks like the local neighborhood I'm in right now." That shortcut usually works, until two different spots in the text look similar, and then the model quietly copies from the wrong place.
Hands-On (~70s, 155 words) The researchers' fix is 2D-RoPE: instead of one street of house numbers, lay the text out on a grid, and give every token two coordinates — a row ID and a column ID, like a spreadsheet cell reference. Copying a passage now means "go to the same row, N columns over" — a fixed, mechanical offset instead of a fuzzy search through a giant 1D sequence. In their experiments, shallow transformers trained with 2D-RoPE copied near-perfectly at input lengths hundreds of times longer than anything they saw during training, while standard 1D-RoPE models fell apart well before that. And it's not just a small-scale trick: they report the same advantage held up when pretraining on the DCLM dataset at sizes up to 1.4 billion parameters, meaning it survives contact with real, large-scale training.
Takeaway (~24s, 55 words) If your product leans on an LLM to quote a contract, restate a log line, or reproduce code exactly, this is worth knowing: the failure isn't prompting, it's the position encoding under the hood. 2D-style positional schemes are a real, testable fix — worth tracking as they move from paper to production models.