arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.02161cs.CVcs.CL
收藏

InfoMerge: 信息感知的令牌压缩用于高效视频大语言模型

InfoMerge: Information-aware Token Compression for Efficient Video Large Language Models

State Key Laboratory of Novel Software Technology(新型软件技术国家重点实验室)

Xinxin Liu, Shiwei Gan, Xiao Liu, Yafeng Yin, Lei Xie, Sanglu Lu

AI总结:

提出InfoMerge,一种无需训练的视觉令牌压缩方法,通过鲁棒冗余估计和内容感知预算分配,在减少85%视觉令牌的同时保持98.8%性能,实现4.24倍预填充加速。

摘要:

视频大语言模型在视频理解中表现出色,但过多的视觉令牌带来了巨大的计算开销。现有的免训练压缩方法通过减少视觉令牌来提高推理效率,但它们通常依赖局部相邻帧相似性进行时间冗余估计,或主要根据片段长度分配令牌预算。这种设计对帧级噪声敏感,且无法捕捉真实视频的非均匀信息分布。为解决这些挑战,我们提出InfoMerge,一种无需训练的视觉令牌压缩方法,通过鲁棒冗余估计和内容感知预算分配来提高令牌利用率。具体来说,我们提出时间指纹差异:一种片段级二阶时间冗余估计策略,用于建模每个片段内相同空间位置令牌的时间相似性结构。我们进一步引入内容感知预算分配(CABA),根据片段独特性和基于谱熵的表征丰富性动态分配片段级令牌预算。通过减少对冗余静态区域的重复保留,并将更多令牌分配给信息丰富的片段,InfoMerge在保持强大性能的同时更好地利用了有限的令牌预算。大量实验表明,InfoMerge在多个基准和骨干网络上实现了强效的精度-效率权衡,在激进压缩下优势更为明显。在LLaVA-OneVision-7B上,InfoMerge保留了原始平均性能的98.8%,同时减少了85%的视觉令牌,并在预填充阶段实现了4.24倍的加速。

英文摘要:

Video Large Language Models (Video-LLMs) achieve strong performance in video understanding, but their excessive visual tokens bring substantial computational overhead. Existing training-free compression methods improve inference efficiency by reducing visual tokens, yet they often rely on local adjacent-frame similarity for temporal redundancy estimation or allocate token budgets mainly according to segment length. Such designs are sensitive to frame-level noise and fail to capture the non-uniform information distribution of real-world videos. To address these challenges, we propose InfoMerge, a training-free visual token compression method that improves token utilization through robust redundancy estimation and content-aware budget allocation. Specifically, we propose the Temporal Fingerprint Difference: a segment-level second-order temporal redundancy estimation strategy, which models the temporal similarity structure of tokens at the same spatial positions within each segment. We further introduce Content-Aware Budget Allocation (CABA), which dynamically allocates segment-level token budgets based on segment uniqueness and spectral-entropy-based representational richness. By reducing repeated preservation of redundant static regions and allocating more tokens to informative segments, InfoMerge makes better use of the limited token budget while maintaining strong performance. Extensive experiments show that InfoMerge achieves strong efficiency--accuracy trade-offs across multiple benchmarks and backbones, with more pronounced advantages under aggressive compression. On LLaVA-OneVision-7B, InfoMerge retains 98.8\% of the original average performance while reducing 85\% of visual tokens and achieving a 4.24-fold speedup in the prefill stage.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑