arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.22158cs.AIcs.CV

ST-SimDiff:平衡时空相似性与差异以实现高效的视频理解与大语言模型

ST-SimDiff: Balancing Spatiotemporal Similarity and Difference for Efficient Video Understanding with MLLMs

Tsinghua University(清华大学);Shenzhen University(深圳大学);Xidian University(西安电子科技大学)

Bingjun Luo, Tony Wang, Chaoqi Chen, Xinpeng Ding

AI总结:

本文提出ST-SimDiff框架,通过平衡时空相似性与差异来提高视频理解效率,利用时空图和双选择策略减少计算成本并提升性能。

摘要:

多模态大语言模型(MLLMs)在处理长视频时面临显著的计算开销,因为需要处理大量的视觉标记。为了提高效率,现有方法主要通过修剪或合并标记来减少冗余,但这些方法忽略了视频内容的一个关键维度,即变化和转折点,并且缺乏对时空关系的协作模型。为此,我们提出了一种新的视角:相似性用于识别冗余,而差异用于捕捉关键事件。基于此,我们设计了一个无需训练的框架,名为ST-SimDiff。我们首先从视觉标记中构建时空图,以统一建模其复杂的关联。随后,我们采用并行双选择策略:1)基于相似性的选择使用社区检测保留代表性标记,压缩静态信息;2)基于时间差异的选择精确定位内容变化点,以保留捕捉关键动态变化的标记。这使它能够用最少的标记保留静态和动态内容。广泛实验表明,我们的方法在显著优于现有最先进方法的同时,大幅减少了计算成本。我们的代码可在https://github.com/bingjunluo/ST-SimDiff上获得。

英文摘要:

Multimodal Large Language Models (MLLMs) face significant computational overhead when processing long videos due to the massive number of visual tokens required. To improve efficiency, existing methods primarily reduce redundancy by pruning or merging tokens based on importance or similarity. However, these approaches largely overlook a critical dimension of video content, i.e., changes and turning points, and they lack a collaborative model for spatio-temporal relationships. To address this, we propose a new perspective: similarity is for identifying redundancy, while difference is for capturing key events. Based on this, we designed a training-free framework named ST-SimDiff. We first construct a spatio-temporal graph from the visual tokens to uniformly model their complex associations. Subsequently, we employ a parallel dual-selection strategy: 1) similarity-based selection uses community detection to retain representative tokens, compressing static information; 2) temporal difference-based selection precisely locates content-changing points to preserve tokens that capture key dynamic shifts. This allows it to preserve both static and dynamic content with a minimal number of tokens. Extensive experiments show our method significantly outperforms state-of-the-art approaches while substantially reducing computational costs. Our code is available in https://github.com/bingjunluo/ST-SimDiff.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑