arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.00512cs.CV
收藏

基于语义边界的小波帧选择用于长视频理解

Wavelet-based Frame Selection by Detecting Semantic Boundary for Long Video Understanding

Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University(中国教育部多媒体可信感知与高效计算重点实验室,厦门大学);College of Computer and Data Science, Fuzhou University(福州大学计算机与数据科学学院)

Wang Chen, Yuhui Zeng, Yongdong Luo, Tianyu Xie, Luojun Lin, Jiayi Ji, Yan Zhang, Xiawu Zheng

AI总结:

本文提出WFS-SB方法,通过小波变换检测语义边界,提升长视频中大视觉语言模型的性能,实验显示在多个基准上均优于现有方法。

摘要:

帧选择在应用大视觉语言模型(LVLM)处理长视频时至关重要,由于高帧冗余和有限的上下文窗口。当前方法通常选择与给定查询相关的高帧,导致不连贯的帧集,忽视视频的叙事结构。本文引入基于小波变换的帧选择(WFS-SB),一种无需训练的框架,提出新的视角:有效的视频理解不仅依赖于高相关性,更重要的是捕捉语义变化——叙事变化的关键时刻,对理解视频整体故事线至关重要。然而,直接检测查询-帧相似度信号中的突变变化往往不可靠,由于模型不确定性和瞬时视觉变化产生的高频噪声。为此,我们利用小波变换,通过其在时间和频率域的多分辨率分析提供理想解决方案。通过应用此变换,我们将噪声信号分解为多个尺度,并从最粗略的尺度中提取干净的语义变化信号。我们识别该信号的局部极值作为语义边界,将视频分割成连贯的片段。在此基础上,WFS-SB包含两个阶段的策略:首先,根据复合重要性分数为每个片段适配性分配帧预算;其次,在每个片段内,采用最大边际相关性方法选择多样且相关的帧集。广泛实验表明,WFS-SB显著提升了LVLM性能,例如在VideoMME上提高准确率5.5%,在MLVU上提高9.5%,在LongVideoBench上提高6.2%,在多个基准上均优于现有方法。我们的代码可在https://github.com/MAC-AutoML/WFS-SB上获得。

英文摘要:

Frame selection is crucial due to high frame redundancy and limited context windows when applying Large Vision-Language Models (LVLMs) to long videos. Current methods typically select frames with high relevance to a given query, resulting in a disjointed set of frames that disregard the narrative structure of video. In this paper, we introduce Wavelet-based Frame Selection by Detecting Semantic Boundary (WFS-SB), a training-free framework that presents a new perspective: effective video understanding hinges not only on high relevance but, more importantly, on capturing semantic shifts - pivotal moments of narrative change that are essential to comprehending the holistic storyline of video. However, direct detection of abrupt changes in the query-frame similarity signal is often unreliable due to high-frequency noise arising from model uncertainty and transient visual variations. To address this, we leverage the wavelet transform, which provides an ideal solution through its multi-resolution analysis in both time and frequency domains. By applying this transform, we decompose the noisy signal into multiple scales and extract a clean semantic change signal from the coarsest scale. We identify the local extrema of this signal as semantic boundaries, which segment the video into coherent clips. Building on this, WFS-SB comprises a two-stage strategy: first, adaptively allocating a frame budget to each clip based on a composite importance score; and second, within each clip, employing the Maximal Marginal Relevance approach to select a diverse yet relevant set of frames. Extensive experiments show that WFS-SB significantly boosts LVLM performance, e.g., improving accuracy by 5.5% on VideoMME, 9.5% on MLVU, and 6.2% on LongVideoBench, consistently outperforming state-of-the-art methods. Our code is available at https://github.com/MAC-AutoML/WFS-SB.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑