Source: arXiv — 2026-08-17
Summary
A new benchmark called Reconstruction tests whether frontier LLMs can actually generate novel research hypotheses, or whether they're just pattern-matching on information leaked into their prompts. It strips a paper down to only the reference list it cited before publication — no title, no abstract, no author names, no post-publication signals — and asks models to guess the paper's core idea from that bibliography alone. Across 643 papers spanning six scientific domains, seven frontier models matched the held-out idea only 3–15% of the time individually. A multi-agent pipeline that ran competing hypotheses through a Swiss-tournament-style bracket improved that to 23–42%, but still missed the majority of ideas.
Key Takeaways
- Reconstruction uses a strict anti-leakage protocol — temporal citation cutoffs, anonymized reference IDs, and frozen per-paper bibliographies — so a model can't infer the idea from anything except which prior papers it cites.
- 643 held-out papers across six scientific domains; a judge LLM scores whether a model's proposed hypothesis matches the actual, undisclosed research idea.
- Individual frontier models: 3–15% match rate from bibliography alone.
- Multi-agent pipeline (each model generates five hypotheses per call, then a Swiss-tournament bracket judges them head-to-head): 23–42% match rate — a ~2.4x improvement, but still failing on the majority of papers.
- The paper's target is the "AI scientist" product category: many of those tools are evaluated with access to full paper text, author identity, or citations that came after publication — exactly the leakage channels Reconstruction is built to close off.
Reel Script
Hook: A wave of "AI scientist" startups claim their models can generate genuine research breakthroughs. A new benchmark just tested that claim honestly — and the best models got it right 3% of the time.
Core Concept: Here's the trick most AI-scientist demos use, intentionally or not: you show the model a paper's title, its citations, sometimes even its abstract, and ask it to "come up with the idea." That's not hypothesis generation, that's reading comprehension. Reconstruction closes that loophole. It gives a model only the bibliography a paper cited — before that paper was even published — strips out anything that could hint at the paper's identity, and asks: from just this reading list, what's the new idea this paper is about to propose? Think of it like handing someone a stack of a detective's case files, minus the detective's own notes, and asking them to solve the case cold.
Hands-On: Across 643 papers in six scientific fields, individual frontier models matched the real idea only 3% to 15% of the time — worse than most people would guess for models this capable. The researchers then tried a multi-agent setup: each model generates five distinct hypotheses in one pass, and instead of picking a winner by vote, the hypotheses get run through a Swiss-tournament bracket — pairwise head-to-head matchups, like a chess tournament, judged by another model — to surface the strongest candidate. That pipeline got the match rate up to 23–42%. Better, but it means even with a tournament of five ideas per model, the majority of real research ideas still went unrecovered.
Takeaway: If your AI-research or "autonomous scientist" pipeline claims high hit rates on novel hypothesis generation, ask what information it actually saw — because on a genuinely blind test, even a tournament of frontier models misses most of the real answer. The honest read: LLMs today are far better at reconstructing ideas from clues than inventing them from nothing, and any product claiming otherwise should show its leakage controls.