arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.23975cs.CVcs.LG
收藏

FrameScout:面向长视频理解的查询相关帧筛选

FrameScout: Scouting Query-Relevant Frames for Long Video Understanding

Southern University of Science and Technology(南方科技大学);The Hong Kong University of Science and Technology(香港科技大学);South China University of Technology(华南理工大学)

Haonan Hu, Shuhao Chen, Weisen Jiang, Lizhao Gao, James Kwok, Yu Zhang

AI总结:

针对长视频视觉令牌过多超出VideoLLMs处理能力的问题,提出带时间上下文编码与帧-查询对比目标的FrameScout,在多基准与模型上实现最优性能。

摘要:

长视频理解要求视频大语言模型(VideoLLMs)回答关于时长数十分钟至数小时的视频的查询,但这类长视频产生的视觉令牌远多于当前VideoLLMs可处理的数量。为解决该问题,关键帧选择器被提出,通过计算帧与查询嵌入的相似度来筛选最相关的帧。现有选择器要么独立编码帧而不利用时间上下文,要么孤立处理片段且采用难以优化的排序目标。为缓解这些局限,我们提出FrameScout,包含两个核心设计:(i)流选择器架构,利用滑动窗口键值缓存与后继聚合生成带时间上下文的帧嵌入;(ii)帧-查询对比目标,在共享空间中对齐帧与查询嵌入,直接训练选择器区分相关与无关帧。在三个长视频基准和六个基础VideoLLMs上的大量实验表明,FrameScout始终达到最优性能,代码可通过此https URL获取。

英文摘要:

Long video understanding requires VideoLLMs to answer queries about videos spanning tens of minutes to hours, but such long videos produce far more visual tokens than current VideoLLMs can process. To address this issue, keyframe selectors are proposed to select the most query-relevant frames by computing the similarity between frame and query embeddings. Existing selectors either encode frames independently without temporal context or process chunks in isolation with ranking objectives that are difficult to optimize. To alleviate those limitations, we propose FrameScout with two core designs: (i) a streaming selector architecture that leverages a sliding-window KV cache and successor aggregation to produce temporally-contextualized frame embeddings; (ii) a frame-query contrastive objective that aligns frame and query embeddings in a shared space, directly training the selector to distinguish relevant frames from irrelevant ones. Extensive experiments on three long-video benchmarks and six base VideoLLMs demonstrate that FrameScout consistently achieves state-of-the-art performance. Code is available at https://github.com/Hubert-1102/FrameScout.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑