arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.04028cs.LGcs.AI
收藏

关于RLVR推理能力边界的争论:收缩、扩张还是两者兼有?一种两阶段动态视角

The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic View

Renmin University of China(中国人民大学);Ant Group(蚂蚁集团);Xiamen University(厦门大学)

Xinhao Yao, Lu Yu, Xiaolin Hu, Fengwei Teng, Qing Cui, Jun Zhou, Yong Liu

AI总结:

该研究提出RLVR推理能力边界变化的两阶段动态理论,调和了能力收缩与扩张的矛盾结论,还为用相对负梯度延长训练、发展高级推理能力提供了理论与实证支撑。

摘要:

关于带可验证奖励的强化学习(reinforcement learning with verifiable rewards, RLVR)究竟是扩展还是收缩大语言模型(large language models, LLMs)推理能力的争论目前仍未得到解决。部分研究主张RLVR主要提升了采样效率,但以牺牲多样性和探索能力为代价,最终导致能力边界收缩。与之相反,另一些研究证明延长训练能够催生全新的推理策略,意味着能力边界的扩张。为调和这些相互矛盾的研究结论,我们从理论和实证层面证明两种观点都具备部分合理性——它们分别对应着一种内在的两阶段概率质量动态过程中的不同阶段:(1)开发(exploitation)阶段:训练初期,模型主要采样已探索过的高奖励和低奖励token,极少选择潜在最优token。正优势估计会提升高奖励token的概率、降低低奖励token的概率,但此阶段内最优token的概率基本保持不变。(2)探索(exploration)阶段:随着训练推进,此前获得的高奖励token概率趋近饱和,增长速率放缓。当潜在最优token——此时已获得正优势估计——被偶尔采样到时,其概率会上升,而原本的高奖励token概率则会下降。这一动态过程表明,开发阶段的过度开发可能导致能力边界收缩,而将训练延长至探索阶段则能够推动推理能力边界扩张。基于我们的研究发现,我们重新探讨了仅使用相对负梯度延长训练的潜力,为更高级推理能力的开发提供了理论与实证基础。

英文摘要:

The ongoing debate on whether reinforcement learning with verifiable rewards (RLVR) expands or shrinks the reasoning capabilities of large language models (LLMs) remains unresolved. Some studies contend that RLVR mainly improves sampling efficiency but at the expense of diversity and exploratory capacity, resulting in capability boundary shrinkage. In contrast, others demonstrate that prolonged training can lead to the emergence of novel reasoning strategies, suggesting capability boundary expansion. To reconcile these contradictory findings, we theoretically and empirically show that both perspectives are partially valid-each aligning with a separate phase in an inherent two-stage probability mass dynamic: (1) Exploitation stage: initially, the model primarily samples explored high-reward and low-reward tokens, while rarely selecting the potentially optimal token. Positive advantage estimates increase the probability of high-reward tokens and decrease those of low-reward tokens, yet the optimal token's probability remains largely unchanged during this stage. (2) Exploration stage: as training advances, the growth rate of previously acquired high-reward tokens slows as their probabilities approach saturation. When a potentially optimal token-now receiving positive advantage estimates-is occasionally sampled, its probability increases, while those of the originally high-reward tokens decrease. This dynamic suggests that over-exploitation during the exploitation stage may lead to capability boundary shrinkage, whereas prolonged training into the exploration stage can promote an expansion of the reasoning capability boundary. Building upon our insights, we revisit the potential of only using relative negative gradients for prolonging training, providing a theoretical and empirical foundation for the development of more advanced reasoning capabilities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑