arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.03584cs.CV
收藏

FrameOracle: 学习在视频中看什么以及看多少

FrameOracle: Learning What to See and How Much to See in Videos

University of Science and Technology of China(中国科学技术大学)

Chaoyu Li, Tianzhi Li, Fei Tao, Zhenyu Zhao, Ziqian Wu, Maozheng Zhao, Juntong Song, Cheng Niu, Pooyan Fazli

AI总结:

提出FrameOracle轻量模块,预测相关帧及其数量,通过课程学习从弱代理信号到强监督,在多个VLM和基准上以更少帧数保持或提升精度。

摘要:

视觉语言模型(VLM)推进了视频理解,但在严格的计算预算下运行,使得性能依赖于选择少量高质量帧子集。现有的帧采样策略,如均匀或固定预算选择,无法适应内容密度或任务复杂性的变化。为了解决这个问题,我们提出了FrameOracle,一个轻量级、即插即用的模块,它同时预测(1)哪些帧与给定查询最相关,以及(2)需要多少帧。FrameOracle通过一个课程进行训练,该课程从弱代理信号(如跨模态相似性)逐步过渡到更强的监督,使用FrameOracle-41K——第一个具有经过验证的关键帧注释(指定每个问题的最小足够帧数)的大规模视频问答数据集。在五个VLM和六个基准上的大量实验表明,FrameOracle将16帧输入减少到平均10.4帧,且无精度损失。当从64帧候选开始时,它将输入平均减少到13.9帧,同时将精度提高1.5%,实现了可扩展视频理解的最优效率-精度权衡。

英文摘要:

Vision-language models (VLMs) advance video understanding but operate under tight computational budgets, making performance dependent on selecting a small, high-quality subset of frames. Existing frame sampling strategies, such as uniform or fixed-budget selection, fail to adapt to variations in content density or task complexity. To address this, we present FrameOracle, a lightweight, plug-and-play module that predicts both (1) which frames are most relevant to a given query and (2) how many frames are needed. FrameOracle is trained via a curriculum that progresses from weak proxy signals, such as cross-modal similarity, to stronger supervision with FrameOracle-41K, the first large-scale VideoQA dataset with validated keyframe annotations specifying minimal sufficient frames per question. Extensive experiments across five VLMs and six benchmarks show that FrameOracle reduces 16-frame inputs to an average of 10.4 frames without accuracy loss. When starting from 64-frame candidates, it reduces inputs to 13.9 frames on average while improving accuracy by 1.5%, achieving state-of-the-art efficiency-accuracy trade-offs for scalable video understanding.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑