Vision-R1: 促进多模态大语言模型推理能力的激励方法
Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models
AI总结：
Vision-R1通过构建高质量多模态CoT数据集和渐进性思维抑制训练策略，提升多模态推理能力，在数学推理基准中取得显著成绩。
摘要：
DeepSeek-R1-Zero 已经成功展示了通过强化学习（RL）在LLM中出现推理能力。受这一突破的启发，我们探索如何利用RL来增强MLLMs的推理能力。然而，直接通过RL训练难以激活MLLMs中的复杂推理能力，如提问和反思，因为缺乏大量高质量的多模态推理数据。为了解决这个问题，我们提出了推理MLLM，即Vision-R1，以提高多模态推理能力。具体来说，我们首先通过利用现有的MLLM和DeepSeek-R1进行模态桥接和数据过滤，构建了一个高质量的多模态CoT数据集，即Vision-R1-cold数据集，用作Vision-R1的冷启动初始化数据。为了缓解冷启动后过度思考带来的优化挑战，我们提出了渐进性思维抑制训练（PTST）策略，并采用基于硬格式结果奖励函数的组相对策略优化（GRPO）来逐步细化模型在10K多模态数学数据集上学习正确和复杂推理过程的能力。全面实验表明，我们的模型在各种多模态数学推理基准上平均提高了约6%。Vision-R1-7B在广泛使用的MathVista基准上实现了73.5%的准确率，仅比领先的推理模型OpenAI O1低0.4%。扩大RL训练中多模态数学数据量，Vision-R1-32B和Vision-R1-72B分别在MathVista基准上获得了76.4%和78.2%的分数。数据集和代码将在：https://github.com/Osilly/Vision-R1 上发布。
英文摘要：
DeepSeek-R1-Zero has successfully demonstrated the emergence of reasoning capabilities in LLMs purely through Reinforcement Learning (RL). Inspired by this breakthrough, we explore how RL can be utilized to enhance the reasoning capability of MLLMs. However, direct training with RL struggles to activate complex reasoning capabilities such as questioning and reflection in MLLMs, due to the absence of substantial high-quality multimodal reasoning data. To address this issue, we propose the reasoning MLLM, Vision-R1, to improve multimodal reasoning capability. Specifically, we first construct a high-quality multimodal CoT dataset without human annotations by leveraging an existing MLLM and DeepSeek-R1 through modality bridging and data filtering to obtain a 200K multimodal CoT dataset, Vision-R1-cold dataset. It serves as cold-start initialization data for Vision-R1. To mitigate the optimization challenges caused by overthinking after cold start, we propose Progressive Thinking Suppression Training (PTST) strategy and employ Group Relative Policy Optimization (GRPO) with the hard formatting result reward function to gradually refine the model's ability to learn correct and complex reasoning processes on a 10K multimodal math dataset. Comprehensive experiments show our model achieves an average improvement of $\sim$6% across various multimodal math reasoning benchmarks. Vision-R1-7B achieves a 73.5% accuracy on the widely used MathVista benchmark, which is only 0.4% lower than the leading reasoning model, OpenAI O1. Scaling up the amount of multimodal math data in the RL training, Vision-R1-32B and Vison-R1-72B achieves 76.4% and 78.2% MathVista benchmark scores, respectively. The datasets and code will be released in: https://github.com/Osilly/Vision-R1 .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
