Source: STAT News — 2026-07-29
Summary
NYU Langone Health researchers pitted frontier general-purpose LLMs (GPT-5.2, Gemini 3.1 Pro, Claude Opus 4.6) against two specialized clinical AI tools built for physicians (OpenEvidence and UpToDate Expert AI). Across three tests — 500 MedQA questions, 500 HealthBench items measuring alignment with clinician preferences, and 100 real de-identified clinical queries physicians had actually asked in practice — the general-purpose models won on all three. The researchers flagged an important caveat: the frontier models were run via clean API access at temperature 0 with search enabled, while the clinical tools were tested through their consumer websites, so the comparison isn't fully controlled.
Key Takeaways
- On MedQA, the general-purpose models scored 97.4% (Gemini 3.1 Pro), 94.2% (GPT-5.2), and 90.2% (Claude Opus 4.6), all beating the specialized clinical tools, which scored 89.6% (OpenEvidence) and 88.4% (UpToDate Expert AI).
- The generalist models also won on HealthBench (clinician-preference alignment) and on a set of 100 real-world de-identified clinical queries, not just on textbook-style questions.
- The tools built specifically for clinicians — designed around trust, citation, and workflow fit — did not out-accuracy the frontier generalist models on their own specialized domain.
- Researchers noted the test wasn't a perfectly controlled head-to-head: frontier models got clean API access at temperature 0 with search enabled, while clinical tools were queried through their regular consumer websites, introducing a prompting-vs-API confound.
Reel Script
Hook (~15-20s, 35-45 words) The AI tools built specifically for doctors just got outscored by ChatGPT-style general models, on medical questions, in the doctors' own specialty. That should make you pause before assuming "purpose-built for healthcare" automatically means more accurate.
Core Concept (~45-90s, 105-200 words) NYU Langone researchers ran a real bake-off: frontier general-purpose models — GPT-5.2, Gemini 3.1 Pro, Claude Opus 4.6 — against two AI tools made specifically for clinicians, OpenEvidence and UpToDate Expert AI. Three tests: a 500-question medical knowledge exam, a 500-item test of whether answers match what clinicians actually want, and 100 real questions doctors had asked an AI system during actual patient care. The generalist models won all three. That's counterintuitive — you'd expect a tool built and tuned specifically for medicine to beat a jack-of-all-trades model. But frontier models today are trained on enormous, broad datasets, and it turns out raw scale and reasoning ability can outrun narrow specialization, at least on these tests. It's a reminder that "specialized for X" is a design choice, not a guarantee of superior performance.
Hands-On (~45-150s, 105-350 words) Here's the actual scoreline, and it matters because the gap isn't small. On the 500-question MedQA test, Gemini 3.1 Pro hit 97.4% accuracy, GPT-5.2 hit 94.2%, and Claude Opus 4.6 hit 90.2%. The two specialized clinical tools came in lower across the board: OpenEvidence at 89.6%, UpToDate Expert AI at 88.4%. So even the weakest generalist model in this test edged out both purpose-built tools. But before you screenshot that and call it settled, there's a catch the researchers were upfront about: this wasn't a fully controlled comparison. The frontier models were accessed cleanly through their APIs, set to temperature zero — meaning deterministic, most-confident answers — with web search turned on. The clinical tools, meanwhile, were tested the way an actual physician would use them: typing queries into the regular consumer website, no special access, no tuned settings. So part of that score gap could be the interface and access method, not just the underlying model quality. That's the nuance that gets stripped out of any headline that just says "AI beats doctors' tools."
Takeaway (~20-30s, 45-70 words) Purpose-built doesn't automatically mean better — frontier general models are closing the gap on specialized tools faster than most people in healthcare IT expected. But don't take the accuracy numbers at face value until you check how each tool was actually queried. Read past the headline, check the methodology section every time.