所有道路通向罗马：激励视觉-语言模型的发散性思维
All Roads Lead to Rome: Incentivizing Divergent Thinking in Vision-Language Models
AI总结：
本文研究了强化学习在视觉-语言模型中激发发散性思维的机制，提出MUPO方法以解决GRPO的多样性崩溃问题，提升模型的多样性和可扩展性。
摘要：
近期研究表明，强化学习（RL），特别是群体相对策略优化（GRPO），能够内在地激发和增强视觉-语言模型（VLMs）的推理能力。然而，尽管有潜力，RL模型的有效性驱动机制及其局限性仍待探索。本文指出RL与基础模型之间存在根本性的行为差异，前者进行更深入但狭窄的推理，而基础模型尽管在个体路径上不够精细，却表现出更广泛和多样化的思维模式。通过进一步分析训练动态，我们发现GRPO容易导致多样性崩溃，导致模型过早收敛到有限的推理策略子集，而丢弃大多数潜在替代方案，导致局部最优和较差的可扩展性。为了解决这一问题，我们提出多组策略优化（MUPO），一种简单而有效的途径，旨在激励多解法的发散性思维，并在已建立的基准上展示了其有效性。项目页面：https://xytian1008.github.io/MUPO/
英文摘要：
Recent studies have demonstrated that Reinforcement Learning (RL), notably Group Relative Policy Optimization (GRPO), can intrinsically elicit and enhance the reasoning capabilities of Vision-Language Models (VLMs). However, despite the promise, the underlying mechanisms that drive the effectiveness of RL models as well as their limitations remain underexplored. In this paper, we highlight a fundamental behavioral distinction between RL and base models, where the former engages in deeper yet narrow reasoning, while base models, despite less refined along individual path, exhibit broader and more diverse thinking patterns. Through further analysis of training dynamics, we show that GRPO is prone to diversity collapse, causing models to prematurely converge to a limited subset of reasoning strategies while discarding the majority of potential alternatives, leading to local optima and poor scalability. To address this, we propose Multi-Group Policy Optimization (MUPO), a simple yet effective approach designed to incentivize divergent thinking across multiple solutions, and demonstrate its effectiveness on established benchmarks. Project page: https://xytian1008.github.io/MUPO/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
