MSJoE：联合进化多模态大语言模型与采样器以实现高效的长视频理解
MSJoE: Jointly Evolving MLLM and Sampler for Efficient Long-Form Video Understanding
AI总结：
MSJoE通过联合进化多模态大语言模型与轻量级采样器，提升长视频理解效率，实验表明其在多个基准测试中表现优异。
摘要：
高效地理解长视频仍然是多模态大语言模型（MLLMs）面临的基本挑战。在本文中，我们提出了MLLM-Sampler联合进化（MSJoE），一种新的框架，该框架联合进化MLLM和一个轻量级的关键帧采样器，以实现高效的长视频理解。MSJoE基于一个关键假设，即只有少量的关键帧对回答每个视频的问题是真正有信息的。具体来说，MSJoE首先推断出几个查询，这些查询描述了与问题相关的多样的视觉视角。然后，这些查询与冻结的CLIP模型交互，生成一个查询帧相似性矩阵。最后，一个轻量级的采样器从该矩阵中预测关键帧采样权重，选择一个紧凑的有信息的帧集，然后将这些帧输入到MLLM中进行答案生成。通过强化学习，MLLM和采样器被联合优化，使查询推理、帧采样和关键帧理解能够协同进化。一个新的长视频问答数据集，包含2800个视频和7000个问题-答案对，被收集以支持训练过程。在VideoMME、LongVideoBench、LVBench和MLVU上进行了广泛的实验，结果表明，MSJoE在基础MLLM上实现了8.0%的准确率提升，并比最强基线方法高1.1%。
英文摘要：
Efficiently understanding long-form videos remains a fundamental challenge for multimodal large language models (MLLMs). In this paper, we present MLLM-Sampler Joint Evolution (MSJoE), a novel framework that jointly evolves the MLLM and a lightweight key-frame sampler for efficient long-form video understanding. MSJoE builds upon a key assumption that only a small subset of key-frames is truly informative for answering each question to a video. Specifically, MSJoE first reasons out several queries, which describe diverse visual perspectives relevant to the question. Then, these queries interact with a frozen CLIP model to produce a query-frame similarity matrix. Finally, a lightweight sampler predicts key-frame sampling weights from this matrix, selecting a compact set of informative frames, which are then fed into the MLLM for answer generation. Both the MLLM and sampler are jointly optimized through reinforcement learning, enabling co-adaptation of query-reasoning, frame-sampling, and key-frame understanding. A new long-video QA dataset containing 2.8K videos with 7K question-answer pairs is collected to support the training process. Extensive experiments on VideoMME, LongVideoBench, LVBench, and MLVU show that MSJoE achieves 8.0\% accuracy gain upon the base MLLM, and 1.1\% higher accuracy than strongest baseline method.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
