arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.01400cs.CV
收藏

通过局部和全局上下文优化实现高效视频大语言模型的token缩减

Token Reduction via Local and Global Contexts Optimization for Efficient Video Large Language Models

University of Trento(特伦托大学);Tsinghua University(清华大学);University of Electronic Science and Technology of China(电子科技大学)

Jinlong Li, Liyuan Jiang, Haonan Zhang, Nicu Sebe

AI总结:

本文提出AOT方法,通过局部-全局最优传输优化视频大语言模型中的token,实现高效的token缩减,保持时间与视觉保真度。

摘要:

视频大语言模型(VLLMs)在视频理解方面表现优异,但因冗余视觉token导致效率低下。现有剪枝方法主要针对帧内空间冗余或在LLM内部进行浅层剪枝,导致时空缩减不充分且未充分利用长上下文压缩性。本文提出一种新视角,通过帧内和帧间token的锚点进行局部-全局最优传输(AOT)以综合聚合信息。具体而言,我们首先在每个帧内建立局部和全局感知的token锚点,然后通过最优传输聚合剪枝token中的信息,构建帧内token锚点。接着,基于时间帧片段,每个片段的第一个帧被视为关键帧锚点,通过最优传输融合连续帧的相似信息,同时保留不同token以表示时间动态,从而在无训练的情况下实现高效的token缩减。大量评估显示,所提AOT方法在各种短视频和长视频基准上均能获得竞争性性能,实现显著的计算效率提升,同时保持时间和视觉保真度。项目网页:https://tyroneli.github.io/AOT。

英文摘要:

Video Large Language Models (VLLMs) demonstrate strong video understanding but suffer from inefficiency due to redundant visual tokens. Existing pruning primary targets intra-frame spatial redundancy or prunes inside the LLM with shallow-layer overhead, yielding suboptimal spatiotemporal reduction and underutilizing long-context compressibility. All of them often discard subtle yet informative context from merged or pruned tokens. In this paper, we propose a new perspective that elaborates token \textbf{A}nchors within intra-frame and inter-frame to comprehensively aggregate the informative contexts via local-global \textbf{O}ptimal \textbf{T}ransport (\textbf{AOT}). Specifically, we first establish local- and global-aware token anchors within each frame under the attention guidance, which then optimal transport aggregates the informative contexts from pruned tokens, constructing intra-frame token anchors. Then, building on the temporal frame clips, the first frame within each clip will be considered as the keyframe anchors to ensemble similar information from consecutive frames through optimal transport, while keeping distinct tokens to represent temporal dynamics, leading to efficient token reduction in a training-free manner. Extensive evaluations show that our proposed AOT obtains competitive performances across various short- and long-video benchmarks on leading video LLMs, obtaining substantial computational efficiency while preserving temporal and visual fidelity. Project webpage: https://tyroneli.github.io/AOT.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑