arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.03615cs.CV
收藏

KTV: 关键帧与关键令牌选择用于高效无训练视频大语言模型

KTV: Keyframes and Key Tokens Selection for Efficient Training-Free Video LLMs

Baiyang Song, Jun Peng, Yuxin Zhang, Guangyao Chen, Feidiao Yang, Jianyuan Guo

AI总结:

KTV通过两阶段框架高效选择关键帧和令牌,提升无训练视频理解的性能和效率。

摘要:

无训练视频理解利用预训练视觉语言模型(VLMs)强大的图像理解能力,将视频视为静态帧的序列,从而避免了昂贵的视频特定训练。然而,这种范式在处理长视频时往往面临严重的视觉冗余和高计算开销。关键帧选择策略,尤其是基于CLIP相似性的策略,容易产生偏差,可能无意中忽略关键帧,导致视频理解不理想。为了解决这些重大挑战,我们提出了KTV,一种新的两阶段框架,用于高效且有效的无训练视频理解。在第一阶段,KTV通过聚类帧级视觉特征进行问题无关的关键帧选择,得到一个紧凑、多样且代表性的帧子集,以减轻时间冗余。在第二阶段,KTV应用关键视觉令牌选择,根据令牌重要性和冗余性从每个选定的关键帧中修剪冗余或信息较少的令牌,这显著减少了输入LLM的令牌数量。在多个选择视频QA任务上的大量实验表明,KTV在使用显著更少的视觉令牌的情况下优于最先进的无训练基线,例如,对于60分钟视频(10800帧),仅使用504个视觉令牌,在MLVU-Test基准上达到44.8%的准确率。特别是,KTV还在某些基准上超过了几种基于训练的方法。

英文摘要:

Training-free video understanding leverages the strong image comprehension capabilities of pre-trained vision language models (VLMs) by treating a video as a sequence of static frames, thus obviating the need for costly video-specific training. However, this paradigm often suffers from severe visual redundancy and high computational overhead, especially when processing long videos. Crucially, existing keyframe selection strategies, especially those based on CLIP similarity, are prone to biases and may inadvertently overlook critical frames, resulting in suboptimal video comprehension. To address these significant challenges, we propose \textbf{KTV}, a novel two-stage framework for efficient and effective training-free video understanding. In the first stage, KTV performs question-agnostic keyframe selection by clustering frame-level visual features, yielding a compact, diverse, and representative subset of frames that mitigates temporal redundancy. In the second stage, KTV applies key visual token selection, pruning redundant or less informative tokens from each selected keyframe based on token importance and redundancy, which significantly reduces the number of tokens fed into the LLM. Extensive experiments on the Multiple-Choice VideoQA task demonstrate that KTV outperforms state-of-the-art training-free baselines while using significantly fewer visual tokens, \emph{e.g.}, only 504 visual tokens for a 60-min video with 10800 frames, achieving $44.8\%$ accuracy on the MLVU-Test benchmark. In particular, KTV also exceeds several training-based approaches on certain benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑