TileMix: Tile-Centric Mixed-Precision Attention for LLM Inference Acceleration

August 19, 2026 | Source: arXiv AI

Tags: LLM inference, quantization, mixed-precision, attention, TileMix, LLaMA, Qwen

TileMix routes LLM attention tiles through FP16 or INT8 at inference time using compact bitmasks — improving long-context prefill throughput over FP16 while recovering quality lost under uniform INT8, tested on LLaMA, Qwen, and Vicuna with no training required.

Details

Long-context prefill is a known bottleneck: dense self-attention scales quadratically in compute and memory. Existing approaches either force uniform lower precision (losing quality) or skip token interactions (losing connectivity). TileMix makes precision a hardware-aligned spatial decision at the tile level within fused dense attention. The method partitions the attention matrix into hardware-aligned tiles and packs routing decisions into compact bitmasks directing each tile group through FP16 or INT8 computation. Both paths share a single online-softmax state, preserving correctness. Routing is training-free and supports grouped-query attention, variable-length batches, and INT8 KV caches. Benchmarks on LongEval, LV-Eval, and A100 prefill throughput tests across LLaMA, Qwen, and Vicuna show TileMix improves throughput over FP16 while outperforming uniform INT8 on quality. Implementation is publicly available. Directly relevant to teams running long-context inference workloads — TileMix could reduce inference cost without the accuracy regression that pure INT8 quantization causes.