面向视觉-语言慢思考推理的半离策略强化学习
Semi-off-Policy Reinforcement Learning for Vision-Language Slow-Thinking Reasoning
AI总结：
针对同策略强化学习受限、离策略强化学习易引发视觉幻觉的问题，提出半离策略强化学习框架SOPHIA，结合同策略视觉理解与离策略推理，有效提升LVLM多模态推理性能，达到开源模型最优水平。
摘要：
增强大型视觉-语言模型（LVLM）的视觉慢思考推理能力，对于解决复杂多模态任务至关重要。然而，由于LVLM主要通过视觉-语言对齐训练，其 rollout 空间受初始能力限制，难以采用同策略强化学习（RL）培养慢思考能力。离策略RL提供了超越当前策略的途径，但直接从外部模型蒸馏轨迹会因不同模型间视觉感知能力不匹配，引发视觉幻觉问题。为解决这些问题，本文提出SOPHIA——一种简单可扩展的面向视觉-语言慢思考推理的半离策略强化学习框架。SOPHIA将可训练LVLM的同策略视觉理解能力，与语言模型的离策略慢思考推理能力相结合，构建半离策略行为模型；为推理分配基于结果的奖励，并反向传播视觉奖励。随后，LVLM利用离策略RL算法，借助传播得到的奖励，从生成的推理轨迹中学习慢思考推理能力。我们在8B和38B规模的InternVL2.5与InternVL3.0模型上开展了大量实验，验证了SOPHIA的有效性。值得注意的是，SOPHIA将InternVL3.0-38B的性能平均提升了8.50%，在多个多模态推理基准上达到开源LVLM的最优性能；在极具挑战性的MathVision和OlympiadBench基准上，其pass@1准确率分别达到49.08%和49.95%，甚至超越了部分闭源模型（如GPT-4.1）。分析表明，SOPHIA的性能优于监督微调和直接同策略RL方法，可为后续同策略训练提供更优的策略初始化。
英文摘要：
Enhancing large vision-language models (LVLMs) with visual slow-thinking reasoning is crucial for solving complex multimodal tasks. However, since LVLMs are mainly trained with vision-language alignment, it is difficult to adopt on-policy reinforcement learning (RL) to develop the slow thinking ability because the rollout space is restricted by its initial abilities. Off-policy RL offers a way to go beyond the current policy, but directly distilling trajectories from external models may cause visual hallucinations due to mismatched visual perception abilities across models. To address these issues, this paper proposes SOPHIA, a simple and scalable Semi-Off-Policy RL for vision-language slow-tHInking reAsoning. SOPHIA builds a semi-off-policy behavior model by combining on-policy visual understanding from a trainable LVLM with off-policy slow-thinking reasoning from a language model, assigns outcome-based rewards to reasoning, and propagates visual rewards backward. Then LVLM learns slow-thinking reasoning ability from the obtained reasoning trajectories using propagated rewards via off-policy RL algorithms. Extensive experiments with InternVL2.5 and InternVL3.0 with 8B and 38B sizes show the effectiveness of SOPHIA. Notably, SOPHIA improves InternVL3.0-38B by 8.50% in average, reaching state-of-the-art performance among open-source LVLMs on multiple multimodal reasoning benchmarks, and even outperforms some closed-source models (e.g., GPT-4.1) on the challenging MathVision and OlympiadBench, achieving 49.08% and 49.95% pass@1 accuracy, respectively. Analysis shows SOPHIA outperforms supervised fine-tuning and direct on-policy RL methods, offering a better policy initialization for further on-policy training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
