AI-Generated Code Correctness: Test the Semantics, Not the Vibe
Testing LLM-generated CUDA and Triton kernels with operator-aware oracles, adversarial inputs, reproducible failures, and false-positive controls.
Direct answer
Direct answer: AI-generated code correctness
Compilation and benchmark speed do not establish that AI-generated code is correct. For numerical kernels, the test oracle must understand the operator’s semantics, generate inputs that expose the likely defect, compare against a trustworthy reference with appropriate tolerances, and preserve a reproducible failure. The evaluator must also be tested against known-correct controls so stronger test inputs do not merely create false positives.
Written for ML systems engineers, GPU programmers, code-generation researchers, evaluation teams, and engineering leaders adopting coding agents.
Architecture
The operating principles
Start from operator semantics
Softmax, normalization, reduction, indexing, masking, and matrix operations fail in different ways. A generic random tensor generator misses operator-specific boundaries.
Use layered oracles
Combine a high-precision CPU reference, invariants, metamorphic properties, shape and stride variation, and implementation inspection rather than trusting one equality check.
Generate adversarial but valid inputs
Boundaries, ties, extreme values, non-contiguous layouts, odd sizes, and degenerate dimensions expose faults that happy-path tensors conceal.
Control false positives
Run the same evaluator against a clean control set and report both recall and false positives. A detector that rejects correct code is not a useful correctness gate.
Preserve the failure artefact
Record the seed, device class, operator, shapes, values, tolerances, generated code, reference output, and environment so the failure can be reproduced.
Evidence
What supports this view
The Correctness Illusion in LLM-Generated GPU Kernels
In the measured corpus, the seeded oracle detected 10/10 injected LLM-style transcription defects, 16/16 correct controls stayed clean, and verdicts matched across five GPU classes.
Inspect the source ↗Test-Input Generation for Tensor Programs
Seven strategies were evaluated on a 26-operator corpus. Boundary-only sampling achieved 78% recall with 0% false positives on 16 controls, while more adversarial values increased both recall and false positives.
Inspect the source ↗gpuemu
Operator-aware correctness tooling and the executable basis for the measured studies.
Inspect the source ↗Scope and limitations
Where the result applies
- —The measured corpus does not estimate the defect rate of any deployed language model or coding product.
- —Both GPU-kernel studies are 2026 arXiv preprints; no peer-reviewed venue is recorded for them.
- —Passing a finite test suite does not prove a generated kernel correct for every possible input or platform.
Common questions
Answers in plain language
Why can an incorrect GPU kernel still look fast?+
A benchmark measures how quickly the implementation runs, not whether it computes the intended function. Skipped work, incorrect indexing, or unstable approximations can make a wrong kernel appear faster.
What is an operator-aware test oracle?+
It is an evaluator that uses the mathematical and shape semantics of the particular operator to generate inputs, choose invariants and references, and interpret numerical tolerances.
Why are random inputs insufficient for tensor-program testing?+
Ordinary random values rarely exercise ties, cancellation, extreme scales, boundary indices, non-contiguous memory, or degenerate dimensions where transcription and optimisation bugs tend to surface.
Does 10/10 defects caught prove all generated kernels are safe?+
No. It establishes performance on a specific seeded corpus. It does not estimate real-world prevalence or prove complete detection.
Continue exploring