Formalizing PTX in Lean to Prove GPU Kernel Correctness

| Source: arXiv AI

Tags: PTX, Lean, Formal Verification, FlashAttention, WGMMA, Mixture of Experts

Researchers formalized NVIDIA's PTX in Lean and used it to prove correctness of simple WGMMA GEMM, FlashAttention and a three-kernel MoE pipeline, bringing formal proofs to deep-learning GPU kernels.

Details

GPU kernels for deep learning rely on tricky patterns such as acquire-release ordering, tensor cores, proxies, async operations and multi-kernel pipelines, where bugs are hard to catch by testing. The paper gives a mathematical formalization of PTX, NVIDIA's GPU intermediate language, implemented in Lean. It models those patterns along with data-dependent control flow. Using it, the authors prove correctness of simple implementations of a WGMMA GEMM kernel, FlashAttention and a three-kernel MoE pipeline. The abstract describes the implementations as simple, so this is an early step rather than verification of production kernels.