PTXBench: Benchmark and Adapt LLMs for GPU Kernel Optimization with Architecture-specific PTX
| Source: arXiv AI
Tags: PTXBench, GPU optimization, LLM fine-tuning, Qwen, NVIDIA, CUDA, benchmarking
PTXBench reveals that no current LLM consistently matches frontier GPU libraries on architecture-specific PTX kernel optimization — success rates on H100/B200 fall from strong on simple GEMM workloads to poor on complex attention backward passes. Fine-tuning Qwen3.6-27B with repair-conditioned training helps but generalizes unevenly.
Details
Writing PTX — NVIDIA's assembly-like intermediate representation — by hand requires deep architecture-specific knowledge that changes with each GPU generation. PTXBench provides the first systematic evaluation of LLMs on this task, measuring functional correctness, runtime execution of target instructions, and speedup versus frontier libraries (cuBLAS, cuDNN) across GEMM and attention workloads on H100 and B200 GPUs.\n\nThe results are clear: no evaluated model consistently beats frontier libraries across the full benchmark suite. PTX capability is uneven — models perform better on simpler GEMM workloads and degrade substantially on complex attention backward passes. Generating the right instructions does not automatically translate into competitive performance, likely because performance optimization requires understanding hardware pipeline interactions beyond mere correctness.\n\nThe team fine-tuned Qwen3.6-27B using supervised fine-tuning with repair-conditioned training, which improves several tasks. However, generalization remains uneven, and data coverage, label balance, and reasoning teacher quality matter more than raw dataset size.\n\nFor infrastructure teams optimizing AI training pipelines, this benchmark provides an honest assessment: LLM-assisted GPU kernel optimization is a real but immature capability. The B200 coverage is timely as new-generation hardware adoption accelerates.