arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.03183cs.CVcs.LG
收藏

MaxInfo: 一种基于最大体积的无训练关键帧选择方法以提升视频理解

MaxInfo: A Training-Free Key-Frame Selection Method Using Maximum Volume for Enhanced Video Understanding

AXXX, Russia(AXXX, 俄罗斯);FusionBrain Lab, Russia(融合脑实验室, 俄罗斯);Institute of Numerical Mathematics, Russia(数值数学研究所, 俄罗斯);Innopolis University, Russia(因诺波利斯大学, 俄罗斯)

Pengyi Li, Irina Abdullaeva, Alexander Gambashidze, Andrey Kuznetsov, Ivan Oseledets

AI总结:

MaxInfo是一种基于最大体积原理的无训练关键帧选择方法,通过提升视频理解性能,有效减少冗余并保持多样性。

摘要:

现代视频大语言模型(VLLMs)通常依赖于均匀帧采样进行视频理解,但这种方法经常无法捕捉到关键信息,因为帧冗余和视频内容的差异性。我们提出了MaxInfo,这是首个基于最大体积原理的无训练方法,它有快速版、慢速版和基于片段的版本,能够从视频中选择并保留最具有代表性的帧。通过最大化所选嵌入形成的几何体积,MaxInfo确保所选帧覆盖嵌入空间中最具有信息量的区域,从而有效减少冗余同时保持多样性。该方法提高了输入表示的质量,并在多个基准上提升了长视频理解性能。例如,MaxInfo在LongVideoBench上实现了3.28%的提升,在EgoSchema上实现了6.4%的提升。此外,MaxInfo在LLaVA-Video-72B上提升了LongVideoBench性能3.47%,在MiniCPM4.5上提升了3.44%。该方法易于实现,可与现有VLLMs配合使用,无需额外训练且具有极低的延迟,使其成为传统均匀采样方法的实用且有效的替代方案。我们的代码可在https://github.com/FusionBrainLab/MaxInfo.git获取。

英文摘要:

Modern Video Large Language Models (VLLMs) often rely on uniform frame sampling for video understanding, but this approach frequently fails to capture critical information due to frame redundancy and variations in video content. We propose MaxInfo, the first training-free method based on the maximum volume principle, which is available in Fast and Slow versions and a Chunk-based version that selects and retains the most representative frames from a video. By maximizing the geometric volume formed by selected embeddings, MaxInfo ensures that the chosen frames cover the most informative regions of the embedding space, effectively reducing redundancy while preserving diversity. This method enhances the quality of input representations and improves long video comprehension performance across benchmarks. For instance, MaxInfo achieves a 3.28% improvement on LongVideoBench and a 6.4% improvement on EgoSchema for LLaVA-Video-7B. Moreover, MaxInfo boosts LongVideoBench performance by 3.47% on LLaVA-Video-72B and 3.44% on MiniCPM4.5. The approach is simple to implement and works with existing VLLMs without the need for additional training and very lower latency, making it a practical and effective alternative to traditional uniform sampling methods. Our code are available at https://github.com/FusionBrainLab/MaxInfo.git

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑