arXiv 2503.243762025-03-31cs.CVcs.AIcs.CLcs.LG

探索强化学习对视频理解的影响：来自 SEED-Bench-R1 的洞察

Exploring the Effect of Reinforcement Learning on Video Understanding: Insights from SEED-Bench-R1

The University of Hong Kong（香港大学）；ARC Lab, Tencent PCG（腾讯平台与内容事业群ARC实验室）；The Chinese University of Hong Kong（香港中文大学）

Yi Chen, Yuying Ge, Rui Wang, Yixiao Ge, Lu Qiu, Ying Shan, Xihui Liu

备注：: Technical Report (In Progress); Code released at: https://github.com/TencentARC/SEED-Bench-R1

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出 SEED-Bench-R1 基准，系统评估强化学习在多模态大模型视频理解中的训练后效果，发现 RL 比监督微调更具数据效率且性能更优，但存在推理链逻辑连贯性不足等问题。

摘要：

思维链（COT）生成的最新进展显著提升了大型语言模型（LLMs）的推理能力，其中强化学习（RL）作为一种有效的训练后方法逐渐兴起。多模态大型语言模型（MLLMs）继承了这种推理潜力，但在需要感知和逻辑推理的任务中仍未得到充分探索。为了解决这个问题，我们引入了 SEED-Bench-R1，这是一个旨在系统评估 MLLMs 在视频理解中训练后方法的基准。它包含复杂的现实世界视频和以多项选择题形式呈现的复杂日常规划任务，需要高级的感知和推理能力。SEED-Bench-R1 通过三级层次结构评估泛化能力：分布内、跨环境和跨环境任务场景，并配备了一个带有易于验证的真实答案的大规模训练数据集。使用 Qwen2-VL-Instruct-7B 作为基础模型，我们将 RL 与监督微调（SFT）进行了比较，证明了 RL 的数据效率及其在分布内和分布外任务上的优越性能，甚至在 LongVideoBench 等通用视频理解基准上也优于 SFT。我们的详细分析表明，RL 增强了视觉感知，但通常会产生逻辑连贯性较差的推理链。我们识别了推理不一致和忽略视觉线索等关键局限性，并建议在基础模型推理、奖励建模以及 RL 对噪声信号的鲁棒性方面进行未来改进。

英文摘要：

Recent advancements in Chain of Thought (COT) generation have significantly improved the reasoning capabilities of Large Language Models (LLMs), with reinforcement learning (RL) emerging as an effective post-training approach. Multimodal Large Language Models (MLLMs) inherit this reasoning potential but remain underexplored in tasks requiring both perception and logical reasoning. To address this, we introduce SEED-Bench-R1, a benchmark designed to systematically evaluate post-training methods for MLLMs in video understanding. It includes intricate real-world videos and complex everyday planning tasks in the format of multiple-choice questions, requiring sophisticated perception and reasoning. SEED-Bench-R1 assesses generalization through a three-level hierarchy: in-distribution, cross-environment, and cross-environment-task scenarios, equipped with a large-scale training dataset with easily verifiable ground-truth answers. Using Qwen2-VL-Instruct-7B as a base model, we compare RL with supervised fine-tuning (SFT), demonstrating RL's data efficiency and superior performance on both in-distribution and out-of-distribution tasks, even outperforming SFT on general video understanding benchmarks like LongVideoBench. Our detailed analysis reveals that RL enhances visual perception but often produces less logically coherent reasoning chains. We identify key limitations such as inconsistent reasoning and overlooked visual cues, and suggest future improvements in base model reasoning, reward modeling, and RL robustness against noisy signals.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。