FlashVID: 通过无训练树状时空标记合并实现高效的视频大语言模型
FlashVID: Efficient Video Large Language Models via Training-free Tree-based Spatiotemporal Token Merging
AI总结：
FlashVID通过无训练的树状时空标记合并技术，在保留99.1%性能的情况下将视频帧输入提升10倍，实现高效视频大语言模型加速。
摘要：
尽管视频大语言模型（VLLMs）在视频理解方面展现了显著的能力，但它们需要处理大量视觉标记，导致显著的计算效率低下。现有的VLLMs加速框架通常独立压缩空间和时间冗余，忽视了时空关系，从而导致次优的时空压缩。由于视频的动态性质，高度相关的视觉特征在时间上可能在空间位置、尺度、方向等属性上发生变化。基于这一见解，我们引入了FlashVID，一种为VLLMs设计的无训练推理加速框架。具体而言，FlashVID利用注意力和多样性基于的标记选择（ADTS）来选择最能代表基本视频表示的标记，然后应用基于树状的时空标记合并（TSTM）进行细粒度的时空冗余消除。在五个视频理解基准测试上对三个代表性VLLMs进行的广泛实验证明了我们方法的有效性和泛化能力。值得注意的是，通过仅保留10%的视觉标记，FlashVID保持了LLaVA-OneVision性能的99.1%。因此，FlashVID可以作为无训练且即插即用的模块，用于扩展长视频帧，这使得视频帧输入能够增加10倍到Qwen2.5-VL，从而在相同计算预算内实现了8.6%的相对提升。代码可在https://github.com/Fanziyang-v/FlashVID上获得。
英文摘要：
Although Video Large Language Models (VLLMs) have shown remarkable capabilities in video understanding, they are required to process high volumes of visual tokens, causing significant computational inefficiency. Existing VLLMs acceleration frameworks usually compress spatial and temporal redundancy independently, which overlooks the spatiotemporal relationships, thereby leading to suboptimal spatiotemporal compression. The highly correlated visual features are likely to change in spatial position, scale, orientation, and other attributes over time due to the dynamic nature of video. Building on this insight, we introduce FlashVID, a training-free inference acceleration framework for VLLMs. Specifically, FlashVID utilizes Attention and Diversity-based Token Selection (ADTS) to select the most representative tokens for basic video representation, then applies Tree-based Spatiotemporal Token Merging (TSTM) for fine-grained spatiotemporal redundancy elimination. Extensive experiments conducted on three representative VLLMs across five video understanding benchmarks demonstrate the effectiveness and generalization of our method. Notably, by retaining only 10% of visual tokens, FlashVID preserves 99.1% of the performance of LLaVA-OneVision. Consequently, FlashVID can serve as a training-free and plug-and-play module for extending long video frames, which enables a 10x increase in video frame input to Qwen2.5-VL, resulting in a relative improvement of 8.6% within the same computational budget. Code is available at https://github.com/Fanziyang-v/FlashVID.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
