A Parametric Linear Conic Projection for Structured Sparse Attention
| Source: arXiv AI
Tags: PLCP, Cone Alignment Index, Structured Sparsity, Sparse Attention, Transformers
A new conic-projection method for structured sparsity produces row, column and diagonal masks suited to speeding up Transformer attention, backed by theory on how many coefficients are pruned per step.
Details
The paper defines the Cone Alignment Index, a scale-invariant sparsity measure based on Lorentz-cone geometry, and analyzes its distribution, showing that at high sparsity samples land inside the cone with probability approaching one. From this it derives two projections: PLCP, a fast closed-form approximation, and PECP, the exact Euclidean projection onto the cone boundary. The authors prove the two are collinear, differ only by a positive scale, and keep the same active support, and they bound how many coefficients are discarded per iteration. A bilevel PLCP framework then generates hardware-friendly structured sparsity patterns aimed at accelerating Transformer attention. The abstract gives no empirical speed or accuracy numbers.