R1-VL：通过逐步组相对策略优化学习多模态大语言模型的推理
R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimization
AI总结：
本文提出StepGRPO在线强化学习框架，通过逐步奖励提升多模态大语言模型的推理能力，并推出R1-VL模型系列，在8个基准上验证了优越性。
摘要：
近期研究通常通过在高质量思维链推理数据上进行监督微调来增强多模态大语言模型（MLLMs）的推理能力，但这往往导致模型仅模仿成功的推理路径，而不理解错误的推理路径是什么。在这项工作中，我们旨在提升MLLMs的推理能力，使其超越被动模仿正向推理路径。为此，我们设计了逐步组相对策略优化（StepGRPO），一种新的在线强化学习框架，通过简单、有效且密集的逐步奖励使MLLMs能够自我提升推理能力。具体而言，StepGRPO引入了两种新颖的基于规则的推理奖励：逐步推理准确性奖励（StepRAR）和逐步推理有效性奖励（StepRVR）。StepRAR通过软关键步骤匹配技术奖励包含必要中间推理步骤的推理路径，而StepRVR通过推理完整性和逻辑评估策略奖励遵循结构良好且逻辑一致的推理过程的推理路径。借助所提出的StepGRPO，我们推出了R1-VL，一系列在逐步推理方面具有出色能力的MLLMs。在8个基准上的大量实验证明了我们方法的优越性。
英文摘要：
Recent studies generally enhance MLLMs' reasoning capabilities via supervised fine-tuning on high-quality chain-of-thought reasoning data, which often leads models to merely imitate successful reasoning paths without understanding what the wrong reasoning paths are. In this work, we aim to enhance the MLLMs' reasoning ability beyond passively imitating positive reasoning paths. To this end, we design Step-wise Group Relative Policy Optimization (StepGRPO), a new online reinforcement learning framework that enables MLLMs to self-improve reasoning ability via simple, effective and dense step-wise rewarding. Specifically, StepGRPO introduces two novel rule-based reasoning rewards: Step-wise Reasoning Accuracy Reward (StepRAR) and Step-wise Reasoning Validity Reward (StepRVR). StepRAR rewards the reasoning paths that contain necessary intermediate reasoning steps via a soft key-step matching technique, while StepRAR rewards reasoning paths that follow a well-structured and logically consistent reasoning process through a reasoning completeness and logic evaluation strategy. With the proposed StepGRPO, we introduce R1-VL, a series of MLLMs with outstanding capabilities in step-by-step reasoning. Extensive experiments over 8 benchmarks demonstrate the superiority of our methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
