arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.20180cs.CVcs.AIcs.CL
收藏

自适应贪心帧选择用于长视频理解

Adaptive Greedy Frame Selection for Long Video Understanding

Purdue University(普渡大学)

Yuning Huang, Xiaoyu Ji, Joseph Huang, Yichi Zhang, Fengqing Zhu

AI总结:

本文提出一种自适应贪心帧选择方法,在固定帧预算下联合优化查询相关性和语义代表性,通过构建1FPS候选池并嵌入两种互补空间,提升长视频问答的准确率。

摘要:

大型视觉-语言模型(VLMs)越来越多地应用于长视频问答,但推理常受限于输入帧数和生成的视觉标记数量。朴素的稀疏采样可能遗漏关键时刻,而纯粹的相关性驱动选择常导致近似重复帧并牺牲时间上远处的证据覆盖。我们提出一种问题自适应的贪心帧选择方法,在固定帧预算下联合优化查询相关性和语义代表性。我们的方法构建一个1~FPS候选池(上限为1000),并精确对齐时间戳,将候选者嵌入两个互补空间(SigLIP用于问题相关性,DINOv2用于语义相似性),并通过贪心最大化相关性项和设施位置覆盖项的加权和来选择帧。该目标被归一化、单调且子模,从而获得标准(1-1/e)的贪心近似保证。为考虑问题依赖的可靠性与覆盖之间的权衡,我们引入四种预设策略和一个轻量级的纯文本问题类型分类器,将每个查询路由到其最佳表现的预设策略。在MLVU上的实验显示,与均匀采样和最近基线相比,在各种帧预算下均实现了稳定的准确性提升,特别是在紧预算下改进最大。

英文摘要:

Large vision--language models (VLMs) are increasingly applied to long-video question answering, yet inference is often bottlenecked by the number of input frames and resulting visual tokens. Naive sparse sampling can miss decisive moments, while purely relevance-driven selection frequently collapses onto near-duplicate frames and sacrifices coverage of temporally distant evidence. We propose a question-adaptive greedy frame selection method that jointly optimizes query relevance and semantic representativeness under a fixed frame budget. Our approach constructs a 1~FPS candidate pool (capped at 1000) with exact timestamp alignment, embeds candidates in two complementary spaces (SigLIP for question relevance and DINOv2 for semantic similarity), and selects frames by greedily maximizing a weighted sum of a modular relevance term and a facility-location coverage term. This objective is normalized, monotone, and submodular, yielding a standard (1-1/e) greedy approximation guarantee. To account for question-dependent trade-offs between relevance and coverage, we introduce four preset strategies and a lightweight text-only question-type classifier that routes each query to its best-performing preset. Experiments on MLVU show consistent accuracy gains over uniform sampling and a strong recent baseline across frame budgets, with the largest improvements under tight budgets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑