arXiv 2505.247182025-06-08cs.CV

以聚焦思考强化视频推理

Reinforcing Video Reasoning with Focused Thinking

Sun Yat-sen University（中山大学）；Lanzhou University（兰州大学）；University of Hong Kong（香港大学）；Hefei University of Technology（合肥工业大学）；National University of Singapore（新加坡国立大学）

Jisheng Dang, Jingze Wu, Teng Wang, Xuanhui Lin, Nannan Zhu, Hongbo Chen, Wei-Shi Zheng, Meng Wang, Tat-Seng Chua

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对视频推理中推理链冗长和二值奖励粗糙的问题，本文提出 TW-GRPO，通过信息熵 token 加权、多选软奖励与问答反转增强实现更高效的强化学习，并在 CLEVRER、MMVU 等基准上取得领先结果。

摘要：

强化学习的最新进展，尤其是通过 Group Relative Policy Optimization（GRPO，组相对策略优化），已显著提升了多模态大语言模型在复杂推理任务上的表现。然而，两个关键局限仍然存在：1）它们往往产生不聚焦、冗长的推理链，掩盖显著的时空线索；2）二值奖励无法考虑部分正确的答案，导致奖励方差高且学习效率低。本文提出 TW-GRPO，这是一种通过聚焦思考和密集奖励粒度增强视觉推理的新颖框架。具体而言，我们采用 token 加权机制，优先考虑具有高信息密度的 token（由组内信息熵估计），抑制诸如通用推理前缀之类的冗余 token。此外，我们通过从单选 QA 任务转向多选 QA 任务来重构 RL 训练，其中软奖励通过区分部分正确性实现更细粒度的梯度估计。另外，我们提出问答反转，这是一种从现有基准生成多样化多选样本的数据增强策略。实验表明，该方法在多个视频推理和通用理解基准上达到最先进性能。值得注意的是，TW-GRPO 在 CLEVRER 上达到 50.4% 的准确率（相比 Video-R1 提升 18.8%），并在 MMVU 上达到 65.8%。代码可在 https://github.com/longmalongma/TW-GRPO 获取。

英文摘要：

Recent advancements in reinforcement learning, particularly through Group Relative Policy Optimization (GRPO), have significantly improved multimodal large language models for complex reasoning tasks. However, two critical limitations persist: 1) they often produce unfocused, verbose reasoning chains that obscure salient spatiotemporal cues and 2) binary rewarding fails to account for partially correct answers, resulting in high reward variance and inefficient learning. In this paper, we propose TW-GRPO, a novel framework that enhances visual reasoning with focused thinking and dense reward granularity. Specifically, we employs a token weighting mechanism that prioritizes tokens with high informational density (estimated by intra-group information entropy), suppressing redundant tokens like generic reasoning prefixes. Furthermore, we reformulate RL training by shifting from single-choice to multi-choice QA tasks, where soft rewards enable finer-grained gradient estimation by distinguishing partial correctness. Additionally, we propose question-answer inversion, a data augmentation strategy to generate diverse multi-choice samples from existing benchmarks. Experiments demonstrate state-of-the-art performance on several video reasoning and general understanding benchmarks. Notably, TW-GRPO achieves 50.4\% accuracy on CLEVRER (18.8\% improvement over Video-R1) and 65.8\% on MMVU. Our codes are available at \href{https://github.com/longmalongma/TW-GRPO}.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。