BtsPruner: Role-Aware Visual Token Pruning for Large Vision-Language Models
| Source: arXiv AI
Tags: BtsPruner, token pruning, LLaVA-1.5, LLaVA-NeXT, Qwen-VL, InternVL
BtsPruner, a training-free visual token pruning method, keeps 91.6% of LLaVA-1.5-7B performance with just 32 of 576 visual tokens and gives up to 5.95x CUDA-latency speedup on LLaVA-NeXT-7B.
Details
Vision-language models often process hundreds or thousands of visual tokens, which drives up compute and memory. Existing training-free pruning treats all tokens with one criterion. BtsPruner observes that tokens shift from local detail carriers to globally referenced representatives during encoding. It finds the transition point from attention concentration, keeps global representatives from later stages, and recursively pulls back local detail tokens from earlier stages under a fixed budget. Tested on LLaVA-1.5, LLaVA-NeXT, Qwen-VL, InternVL and CogVLM, it keeps 91.6% of dense performance on LLaVA-1.5-7B with 5.6% of tokens and reaches up to 5.95x CUDA-latency speedup in high-resolution LLaVA-NeXT-7B.