Source: arXiv — 2026-08-13
Summary
Rishi Shah and Rishav Shrestha of E3A Healthcare introduce a 12-gate, largely tolerance-free verifier designed to catch silent failures in LLM-generated GPU kernels — NaN/Inf mishandling, run-to-run nondeterminism, shape fragility, and precision drift — that ordinary "run it and see" correctness checks miss. Running it against 2,638 machine-generated kernels a public system had already accepted as correct, they found 39.5% broken outright and 62.1% carrying at least one contract violation. They then use the verifier to validate a new, natively-written Blackwell tcgen05 training backward pass for the gated-linear-recurrence model family.
Key Takeaways
- Standard correctness checks for AI-generated GPU kernels typically run a few random inputs and compare outputs within a numerical tolerance — a method that misses failures that only show up under specific conditions.
- The verifier's 12 gates are largely tolerance-free: a kernel either satisfies a hard property (no silent NaN substitution, deterministic across runs, stable under shape changes, no unexplained precision loss) or it fails — no threshold to explain away a near-miss.
- Applied retroactively to 2,638 kernels a public system had already labeled "correct," 39.5% failed outright and 62.1% had at least one contract violation — meaning a majority of "verified" kernels were quietly untrustworthy.
- The paper also delivers a genuinely new artifact: the first native Blackwell tcgen05 training backward pass for the gated-linear-recurrence (GDN) model family, checked independently against a double-precision numerical oracle and used to train five family members.
- This is a direct warning for anyone trusting an AI system's self-reported "tests passed" on generated low-level code — the tests themselves need auditing, not just the code.
Reel Script
Hook: A system had already marked 2,638 AI-generated GPU kernels as correct. When researchers built a stricter verifier and re-checked every one, nearly forty percent turned out to be silently broken — and the tests that missed it are the same kind almost every AI coding tool relies on.
Core Concept: A GPU kernel is a tiny, low-level piece of code that does one specific math operation as fast as possible on the graphics chip — the atomic building block underneath every large AI model's training run. When you ask an AI to generate one of these, the usual way to check it's correct is to run it on a handful of random inputs and compare the output against a reference, allowing some small numerical wiggle room. That sounds reasonable, but it misses an entire category of failure: a kernel that quietly returns an ordinary number instead of crashing when the math actually produces NaN or infinity, one that gives a different answer every time you rerun it because of a race condition, one that only breaks once you feed it a slightly different input shape, or one that slowly loses precision because it's accumulating results in a smaller number format than the reference. None of those show up if you just spot-check a few outputs within a tolerance band — they need dedicated, tolerance-free checks, which is exactly what this paper builds: twelve separate gates, each one a hard pass-or-fail property instead of a fuzzy threshold.
Hands-On: Here's the number worth putting on screen: out of 2,638 kernels that a separate system had already accepted as correct, running them back through these twelve gates found 39.5% were broken beyond any reasonable tolerance argument, and 62.1% — nearly two-thirds — violated at least one of the twelve contracts. That's the gap between "passed a random spot-check" and "actually correct," made visible for the first time at scale. The second half of the paper puts the verifier to real use: the authors hand-write the first native Blackwell tcgen05 training backward pass for the gated-linear-recurrence family of models — a genuinely new, hard piece of low-level GPU code — and instead of trusting their own tests, they check its correctness independently against a double-precision numerical oracle before training five real models through it.
Takeaway: If any part of your stack trusts an AI system's self-reported "tests passed" on generated low-level or performance-critical code, treat that claim as unverified until you know what the tests actually check for. Tolerance-based spot-checks are not enough at this level — go build or borrow gates that check hard invariants, not averages.