arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.00983cs.CV
收藏

基于事件的帧选择用于有效长视频理解

Event-Anchored Frame Selection for Effective Long-Video Understanding

Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University(教育部多媒体可信感知与高效计算重点实验室,厦门大学);College of Computer and Data Science, Fuzhou University(福州大学计算机与数据科学学院)

Wang Chen, Yongdong Luo, Yuhui Zeng, Luojun Lin, Tianyu Xie, Fei Chao, Rongrong Ji, Xiawu Zheng

AI总结:

本文提出基于事件的帧选择方法,通过分层事件感知流程提升长视频理解的效率和效果。

摘要:

大规模帧冗余和有限的上下文窗口使得在大视觉-语言模型(LVLMs)中高效选择帧对于长视频理解至关重要。然而,现有方法却采用了一种扁平采样范式,将视频视为无结构的帧集合。在本文中,我们引入了事件锚定帧选择(EFS),一种分层、事件感知的流程。利用自监督DINO嵌入,EFS首先将视频流划分为视觉上同质的时间段,这些时间段作为语义事件的代理。在每个事件内,它然后选择最相关的查询帧作为锚点。这些锚点作为结构先验,引导使用自适应最大边际相关性(MMR)方案的全局细化阶段。该流程确保最终的关键帧集共同优化事件覆盖、查询相关性和视觉多样性。作为一种无需训练的即插即用模块,EFS可以无缝集成到现成的LVLMs中,在具有挑战性的视频理解基准上取得显著收益。具体而言,当应用于LLaVA-Video-7B时,EFS在VideoMME、LongVideoBench和MLVU上分别提高了准确率4.7%、4.9%和8.8%。

英文摘要:

Massive frame redundancy and limited context window make efficient frame selection crucial for long-video understanding with large vision-language models (LVLMs). Prevailing approaches, however, adopt a flat sampling paradigm which treats the video as an unstructured collection of frames. In this paper, we introduce Event-Anchored Frame Selection (EFS), a hierarchical, event-aware pipeline. Leveraging self-supervised DINO embeddings, EFS first partitions the video stream into visually homogeneous temporal segments, which serve as proxies for semantic events. Within each event, it then selects the most query-relevant frame as an anchor. These anchors act as structural priors that guide a global refinement stage using an adaptive Maximal Marginal Relevance (MMR) scheme. This pipeline ensures the final keyframe set jointly optimizes for event coverage, query relevance, and visual diversity. As a training-free, plug-and-play module, EFS can be seamlessly integrated into off-the-shelf LVLMs, yielding substantial gains on challenging video understanding benchmarks. Specifically, when applied to LLaVA-Video-7B, EFS improves accuracy by 4.7%, 4.9%, and 8.8% on VideoMME, LongVideoBench, and MLVU, respectively.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑