arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.19680cs.CVcs.AI
收藏

基于M-LLM的视频帧选择用于高效视频理解

M-LLM Based Video Frame Selection for Efficient Video Understanding

Carnegie Mellon University(卡内基梅隆大学);University of Central Florida(中佛罗里达大学);Amazon(亚马逊公司)

Kai Hu, Feng Gao, Xiaohan Nie, Peng Zhou, Son Tran, Tal Neiman, Lingyun Wang, Mubarak Shah, Raffay Hamid, Bing Yin, Trishul Chilimbi

AI总结:

针对视频理解中均匀采样丢失关键帧的问题,提出基于轻量级M-LLM的自适应帧选择方法,利用空间和时间监督信号训练选择器,提升下游video-LLM在多个视频问答基准上的性能。

摘要:

近年来,多模态大语言模型(M-LLMs)在视频推理方面展现出令人鼓舞的成果。流行的多模态大语言模型(M-LLM)框架通常采用朴素的均匀采样来减少输入到M-LLM中的视频帧数量,特别是对于长上下文视频。然而,这种方法可能会丢失视频中某些时间段的关键上下文,导致下游M-LLM可能没有足够的视觉信息来回答问题。为了解决这一痛点,我们提出了一种基于轻量级M-LLM的帧选择方法,能够自适应地选择与用户查询更相关的帧。为了训练所提出的帧选择器,我们引入了两种监督信号:(i)空间信号,通过提示M-LLM获得单帧重要性分数;(ii)时间信号,通过使用所有帧候选的标题提示大语言模型(LLM)来选择多帧。随后,选中的帧被一个冻结的下游视频M-LLM消化,用于视觉推理和问答。实证结果表明,所提出的M-LLM视频帧选择器在中等长度(ActivityNet、NExT-QA)和长上下文(EgoSchema、LongVideoBench)视频问答基准上,提升了各种下游视频大语言模型(video-LLM)的性能。

英文摘要:

Recent advances in Multi-Modal Large Language Models (M-LLMs) show promising results in video reasoning. Popular Multi-Modal Large Language Model (M-LLM) frameworks usually apply naive uniform sampling to reduce the number of video frames that are fed into an M-LLM, particularly for long context videos. However, it could lose crucial context in certain periods of a video, so that the downstream M-LLM may not have sufficient visual information to answer a question. To attack this pain point, we propose a light-weight M-LLM -based frame selection method that adaptively select frames that are more relevant to users' queries. In order to train the proposed frame selector, we introduce two supervision signals (i) Spatial signal, where single frame importance score by prompting a M-LLM; (ii) Temporal signal, in which multiple frames selection by prompting Large Language Model (LLM) using the captions of all frame candidates. The selected frames are then digested by a frozen downstream video M-LLM for visual reasoning and question answering. Empirical results show that the proposed M-LLM video frame selector improves the performances various downstream video Large Language Model (video-LLM) across medium (ActivityNet, NExT-QA) and long (EgoSchema, LongVideoBench) context video question answering benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑