arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2512.11534cs.CVcs.CLcs.MM
收藏

HFS: 为高效视频推理的全局查询感知帧选择

HFS: Holistic Query-Aware Frame Selection for Efficient Video Understanding

The Hong Kong Polytechnic University(香港理工大学)

Yiqing Yang, Yun Li, Daiqing Qi, Lehan Yang, Tianlong Wang, Wenhao Zhang, Sheng Li, Kin-man Lam

AI总结:

HFS提出一种端到端可训练的帧选择框架,通过任务自适应方法提升视频推理效率。

摘要:

视频理解中的关键帧选择面临重大挑战。传统top-K选择方法,通过独立评分帧,往往无法整体优化选择。这种独立评分常导致选择的帧在时间上聚集且视觉冗余。此外,使用多模态大语言模型(MLLMs)生成的伪标签训练轻量级选择器,会阻止监督信号动态适应任务目标。为解决这些限制,我们提出了一种端到端可训练、任务自适应的帧选择框架。一种链式思维方法指导小型语言模型(SLM)生成任务特定的隐式查询向量,这些向量与多模态特征结合,以实现动态帧评分。我们进一步定义了一个连续的集级别目标函数,该函数结合了相关性、覆盖性和冗余性,通过Gumbel-Softmax实现可微优化,以在集级别选择最佳帧组合。最后,采用学生-教师互学习方法,其中学生选择器(SLM)和教师推理器(MLLM)通过KL散度对齐帧重要性分布。结合交叉熵损失,这使得能够实现端到端优化,摆脱对静态伪标签的依赖。在各种基准测试中,包括Video-MME、LongVideoBench、MLVU和NExT-QA,实验表明,我们的方法显著优于现有方法。

英文摘要:

Key frame selection is essentially a set-level optimization problem: the quality of the selected subset depends on the interactions among frames, rather than the score of any single frame. Existing methods generally exhibit three major limitations. Point-wise methods score each frame independently and ignore inter-frame dependencies. Although the training-free set-level methods explicitly model the inter-frame relationships, their selection criteria are fixed and cannot be adapted through downstream task feedback. Learnable methods can leverage data-driven training; however, they lack an explicit, differentiable set-quality objective and rely on offline-generated supervision signals. To address these limitations, we propose an end-to-end trainable and task-adaptive framework for frame selection. A Chain-of-Thought prompt conditions a Small Language Model (SLM) to extract task-specific latent query vectors, which are combined with multimodal features to enable dynamic, query-aware frame scoring. We further formulate a continuous set-level objective function that jointly accounts for relevance, coverage, and redundancy, enabling differentiable set-level optimization via Gumbel-TopK for selecting optimal frame combinations. Finally, we employ a student-teacher mutual learning strategy, in which the student selector (SLM) and teacher reasoner (MLLM) are trained to align their frame-importance distributions via KL divergence. Combined with cross-entropy loss, this design enables fully end-to-end optimization, eliminating reliance on static pseudo-labels. Experiments across multiple benchmarks, including Video-MME, LongVideoBench, MLVU, and NExT-QA, demonstrate that our method significantly outperforms existing frame-selection approaches.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑