强化结构链式思考以提升视频理解
Reinforcing Structured Chain-of-Thought for Video Understanding
AI总结：
本文提出SDRL框架，通过结构化链式思考格式和自监督机制，解决多模态大语言模型在视频理解中的推理漂移和时间感知问题，实现单阶段强化学习，提升模型泛化能力。
摘要：
多模态大语言模型（MLLMs）在视频理解中展现出潜力，但其推理常面临思维漂移和时间感知薄弱的问题，即使通过强化学习（RL）技术如分组相对策略优化（GRPO）增强后仍难以解决。现有RL方法通常依赖监督微调（SFT），需要昂贵的链式思考（CoT）标注和多阶段训练，且强制固定推理路径，限制了MLLMs的泛化能力并可能引入偏差。为克服这些限制，我们引入了基于摘要的强化学习（SDRL），一种新颖的单阶段RL框架，通过使用结构化CoT格式：总结->思考->回答，消除了SFT的必要性。SDRL在GRPO目标中引入了两种自监督机制：1）视觉知识一致性（CVK）通过减少生成摘要间的KL散度来强制事实基础；2）推理动态多样性（DVR）通过根据小组准确性动态调节思考多样性来促进探索。这种新颖的整合有效平衡了对齐和探索，监督最终答案和推理过程。我们的方法在七个公共VideoQA数据集上实现了最先进的性能。
英文摘要：
Multi-modal Large Language Models (MLLMs) show promise in video understanding. However, their reasoning often suffers from thinking drift and weak temporal comprehension, even when enhanced by Reinforcement Learning (RL) techniques like Group Relative Policy Optimization (GRPO). Moreover, existing RL methods usually depend on Supervised Fine-Tuning (SFT), which requires costly Chain-of-Thought (CoT) annotation and multi-stage training, and enforces fixed reasoning paths, limiting MLLMs' ability to generalize and potentially inducing bias. To overcome these limitations, we introduce Summary-Driven Reinforcement Learning (SDRL), a novel single-stage RL framework that obviates the need for SFT by utilizing a Structured CoT format: Summarize -> Think -> Answer. SDRL introduces two self-supervised mechanisms integrated into the GRPO objective: 1) Consistency of Vision Knowledge (CVK) enforces factual grounding by reducing KL divergence among generated summaries; and 2) Dynamic Variety of Reasoning (DVR) promotes exploration by dynamically modulating thinking diversity based on group accuracy. This novel integration effectively balances alignment and exploration, supervising both the final answer and the reasoning process. Our method achieves state-of-the-art performance on seven public VideoQA datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
