Source: arXiv — 2026-08-12
Summary
A case study by independent AI engineer Joel Abenhaim documents an AI coding agent, working under a "specification-first" protocol, dismantling a core architectural invariant across 189 files in a 717,725-line, 3,648-file production TypeScript codebase. The old rule was that a UI panel had to stay open for the duration of an AI request; the agent rewrote the system so a streaming generation survives the panel being closed and correctly reattaches when reopened — with no pre-existing test defining correct behavior and no human code review gating the change.
Key Takeaways
- The protocol locks a written specification before any code changes, and the agent's job is to converge the codebase on that spec rather than iterate ad hoc against vague instructions.
- The invariant being replaced spanned the entire application: "a panel must stay open for a request to live" became "a stream survives panel close and re-opens cleanly, without loss or duplication."
- 189 files changed in a 3,648-file, 717k-line production codebase — a scale of coordinated refactor that would typically require a dedicated team and a multi-sprint migration plan.
- No test oracle existed to define "correct" ahead of time, and no human reviewed the diff before merge — success was judged entirely by the agent's own convergence against the locked spec.
- This is a live data point on how far autonomous-agent-driven refactoring can go in a real, large, imperfect codebase, not a toy benchmark repo.
Reel Script
Hook: An AI agent just rewrote a load-bearing rule that touched 189 files in a real production codebase — with no tests to check its work and nobody reviewing the diff before it shipped. That should terrify you a little, and it's worth understanding exactly why it didn't fail.
Core Concept: Most AI coding agent stories are about small, contained changes — fix this bug, add this endpoint. This one is different because it's about an invariant: a rule baked so deep into a codebase's architecture that changing it ripples everywhere. The old invariant here was "a UI panel has to stay open for an AI request tied to it to keep running" — close the panel, the request effectively dies. The new invariant is "a streaming response survives the panel closing, and reattaches cleanly if you reopen it, without losing or duplicating any of the output." That's not a bug fix, that's rewriting an assumption the entire application was built around. The method that made it survivable is called specification-first: instead of the agent freelancing toward a goal, a human locks a written spec describing the target behavior precisely, and the agent's entire job becomes converging the actual code toward matching that spec — repeatedly checking its own work against a fixed target instead of guessing what "done" means.
Hands-On: Picture the scale here: a 717,725-line codebase spread across 3,648 files, and the agent touched 189 of them to pull off this one invariant swap — that's roughly one in twenty files in the entire project, coordinated correctly enough that a stream can now survive a panel closing and reopening without duplicating a single output token. Normally that's the kind of refactor a company puts on a roadmap with a dedicated team and a staged rollout plan spanning weeks. What's worth sketching out is the workflow itself: spec gets written and locked first, agent makes a pass, agent checks its own output against the spec, agent iterates — repeat until the actual code state matches the locked target, with zero pre-existing test suite defining "correct" and zero human eyeballing the diff along the way. The convergence loop against a fixed spec is doing the job a test suite or a reviewer would normally do.
Takeaway: If you're evaluating whether agentic coding tools are ready for anything beyond toy tasks, this case study is your evidence that large, cross-cutting refactors are now in reach — but notice what made it work: a precisely locked specification, not a vague prompt. Don't hand an agent a fuzzy goal and walk away; hand it a spec this tight, or don't trust the result.