Coarse Semantic Blocks for Efficient Training-Free Sparse Attention

| Source: arXiv AI

Tags: Sparse Attention, Long Context, Qwen3.5, NVIDIA H200, RULER, LongBench-v2

A training-free sparse attention method that clusters keys into large semantic blocks delivers 2.3-2.4x prefill and 1.7-2.0x decode speedups at 512K tokens on Qwen3.5 models while staying close to dense accuracy for the 35B-A3B model.

Details

Long-context inference cost is dominated by attention. This method builds a prompt-specific key index by clustering keys into large semantic blocks that group tokens from noncontiguous positions. Each query scores block centroids and attends to original keys and values inside chosen blocks, and queries selecting the same block are processed together. The authors argue larger blocks with a larger token budget can be both faster and more accurate than smaller ones. On one NVIDIA H200, including index costs, whole-model speedups over dense attention were 2.3-2.4x for prefill and 1.7-2.0x for decoding at 512K, and 2.9-3.0x for prefill at 1M. Qwen3.5-35B-A3B stayed close to dense attention on five RULER tasks through 1M, on LongBench-v2 and on MRCR through 256K. The 4B model lost accuracy mainly on multi-key retrieval.