MixQuant: Adaptive Mixed-Precision Quantization for Large Language Models

July 28, 2026 | Source: arXiv AI

Tags: quantization, Llama, Mistral, AWQ, GPTQ, mixed-precision, LLM inference

MixQuant improves adaptive LLM quantization by accounting for upstream layer dependencies when scoring sensitivity, improving average accuracy by up to 8 points and reducing perplexity from 12.43 to 10.70 on Llama and Mistral models at the tightest memory budgets.

Details

Mixed-precision quantization allocates higher bitwidths to sensitive layers, but existing methods score layer sensitivity assuming other layers are full-precision—which is wrong in practice. MixQuant addresses this by marginalizing each layer's distortion over random quantized upstream configurations, giving budget-agnostic sensitivity scores that reflect true inter-layer dependencies. The framework is technique-agnostic, wrapping any base quantizer (AWQ or GPTQ). A single calibration pass serves any deployment budget, and a final greedy allocation runs at negligible cost. Allocations are also penalized if they leave layers at the lowest bitwidths, preventing degenerate solutions. Tested on Llama-3.2-3B, Llama-2-7B, and Mistral-7B across AWQ and GPTQ, MixQuant outperforms adaptive and mixed-precision baselines in every setting—up to 8 points average accuracy improvement and perplexity reduction from 12.43 to 10.70 at the tightest memory budget. It matches ILP solver quality at negligible deployment overhead. For teams running quantized inference on edge hardware or cost-sensitive cloud deployments, MixQuant is a drop-in improvement over standard adaptive quantization requiring no retraining and working with existing quantizers.