arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.01002cs.CVcs.AIcs.LG

基于证据的关键帧采样用于基于MLLM的长视频理解

Query-Conditioned Evidential Keyframe Sampling for MLLM-Based Long-Form Video Understanding

Duke University(杜克大学);Independent Researcher(独立研究员)

Yiheng Wang, Lichen Zhu, Yueqian Lin, Yudong Liu, Jingyang Zhang, Hai "Helen" Li, Yiran Chen

AI总结:

本文提出基于信息瓶颈理论的证据驱动关键帧采样框架,通过最大化选帧与查询的条件互信息,提升长视频理解性能,实验表明在严格token预算下优于现有方法。

摘要:

多模态大语言模型(MLLMs)在视频问答中表现优异,但应用于长视频受限于上下文长度和计算成本,关键帧采样至关重要。现有方法通常依赖语义相关性或强化学习,难以捕捉证据线索或存在低效的组合优化。本文提出基于信息瓶颈理论的证据驱动关键帧采样框架,将关键帧选择建模为最大化选帧与查询的条件互信息,提供一个反映各帧回答问题贡献的原理性目标。为使该目标可 tractable,我们利用其结构推导出分解优化,将子集选择转化为独立帧级评分。进一步引入一个基于对比目标训练的查询条件证据评分网络,高效估计证据重要性。在长视频理解基准上的实验表明,我们的方法在严格token预算下持续优于现有采样策略,同时显著提高训练效率。

英文摘要:

Multimodal Large Language Models (MLLMs) have shown strong performance on video question answering, but their application to long-form videos is constrained by limited context length and computational cost, making keyframe sampling essential. Existing approaches typically rely on semantic relevance or reinforcement learning, which either fail to capture evidential clues or suffer from inefficient combinatorial optimization. In this work, we propose an evidence-driven keyframe sampling framework grounded in information bottleneck theory. We formulate keyframe selection as maximizing the conditional mutual information between selected frames and the query, providing a principled objective that reflects each frame's contribution to answering the question. To make this objective tractable, we exploit its structure to derive a decomposed optimization that reduces subset selection to independent frame-level scoring. We further introduce a query-conditioned evidence scoring network trained with a contrastive objective to estimate evidential importance efficiently. Experiments on long-form video understanding benchmarks show that our method consistently outperforms prior sampling strategies under strict token budgets, while significantly improving training efficiency.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑