INFRA Signal 523
Contract-grade verifier finds 39.5% of accepted LLM-generated GPU kernels broken beyond tolerance
↗
Illustration only Photo by Vishnu Mohanan on Unsplash
A twelve-gate contract-grade verifier for LLM-generated GPU kernels found that 39.5% of 2,638 machine-generated kernels previously accepted as correct were broken beyond any tolerance argument, with 62.1% carrying at least one violation.
Engineers relying on LLM-generated GPU kernels that pass standard correctness tests may be shipping silently incorrect code, since the field's common test only checks a few random inputs at one fixed shape. A contract-grade verifier with tolerance-free gates provides a much stronger correctness signal that could prevent subtle numerical bugs from reaching production.
Written by elseif from the cluster below · every claim links back to a sourceThe three things worth knowing
The verifier's twelve adversarial gates include several tolerance-free checks that no threshold choice can explain away, catching issues like NaN masking, non-determinism, shape sensitivity, and precision mismatch.
Of 2,638 kernels a public system had already accepted, the standard test passed 1,487 that the verifier rejected, while only 14 passed the standard test but failed the verifier.
The finding was defended through four independent methods: a 7/7 positive control, threshold-calibration sweep, 98.5% agreement with a reference benchmark's own correctness code, and a stratified hand-audit.
THE CLUSTER