面向长视频混合视觉语言模型的有状态令牌缩减
Stateful Token Reduction for Long-Video Hybrid VLMs
AI总结：
针对混合Mamba-Transformer视觉语言模型，提出有状态令牌缩减方法，利用Mamba层的状态保持特性实现压缩而非剪枝，在25%令牌预算下实现3.8-4.2倍预填充加速，同时保持基线精度。
摘要：
令牌缩减加速了长视频视觉语言模型（VLM），但现有方法针对Transformer，将缩减视为令牌剪枝。我们研究混合Mamba-Transformer VLM中的令牌缩减，发现它是\textit{有状态的}：Mamba层维护一个从早期令牌累积信息的循环状态，使丢弃的令牌得以保留，因此缩减更像压缩而非剪枝。我们通过基于表示的概率方法支持这一观点，测量丢弃令牌信息的保留程度，并分析逐层稀疏性和跨层重要性稳定性。我们的发现表明，重要性在层内稀疏但跨层不稳定，使得激进的早期剪枝不可靠，而混合模型对后期缩减保持鲁棒。受此启发，我们提出一个混合感知的令牌缩减框架，包含从低到高的渐进式调度和用于注意力与Mamba层的统一查询条件重要性分数。对于Mamba，排除循环中与位置相关的衰减可产生更强的选择信号。在长视频基准测试中，我们的方法在25%令牌预算下实现了3.8倍至4.2倍的预填充加速，同时保持接近基线的精度，并通过轻量微调进一步提升。混合模型受益于激进的缩减，提高了效率和精度，而Transformer则表现出标准的权衡。我们的方法在相同混合骨干网络上优于先前基线，并与视觉冗余缩减方法有效结合。
英文摘要：
Token reduction accelerates long-video vision--language models (VLMs), but existing methods target Transformers, where reduction is treated as token pruning. We study token reduction in hybrid Mamba--Transformer VLMs and find that it is \emph{stateful}: Mamba layers maintain a recurrent state that accumulates information from earlier tokens, allowing discarded tokens to persist, so reduction behaves more like compression than dropping.We support this view with a representation-based probing method measuring how much information from discarded tokens is retained, and analyze layer-wise sparsity and cross-layer importance stability. Our findings show importance is sparse within layers but unstable across layers, making aggressive early pruning unreliable while hybrids remain robust to later reduction.Motivated by this, we propose a hybrid-aware token reduction framework with a low-to-high progressive schedule and a unified query-conditioned importance score for attention and Mamba layers. For Mamba, excluding the position-dependent decay from the recurrence produces a stronger selection signal. Across long-video benchmarks, our method achieves $3.8{\times}$--$4.2{\times}$ prefilling speedups at a 25% token budget while maintaining near-baseline accuracy and improving with light finetuning. Hybrid models benefit from aggressive reduction, improving both efficiency and accuracy, whereas Transformers exhibit the standard trade-off. Our method also outperforms prior baselines on the same hybrid backbone and combines effectively with visual redundancy reduction methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
