长度无偏序列策略优化:揭示和控制RLVR中的响应长度变化
Length-Unbiased Sequence Policy Optimization: Revealing and Controlling Response Length Variation in RLVR
AI总结:
本文提出LUSPO算法,通过消除长度偏差解决RLVR中响应长度崩溃问题,并在多个任务中展示出优于现有方法的性能。
摘要:
最近将可验证奖励的强化学习(RLVR)应用于大型语言模型(LLMs)和视觉语言模型(VLMs)的实践,已证明在复杂任务中增强推理能力方面取得了显著成功。在RLVR训练过程中,响应长度的增加通常被视为推动推理能力增长的关键因素。然而,不同RLVR算法在训练过程中响应长度变化的模式差异显著。为了为这些差异提供基本解释,本文对主流RLVR算法的组成部分进行了深入分析。我们对影响响应长度的因素进行了理论分析,并通过大量实验验证了我们的理论。基于这些理论发现,我们提出了长度无偏序列策略优化(LUSPO)算法。具体而言,我们纠正了群序列策略优化(GSPO)中固有的长度偏差,使损失函数对响应长度无偏,从而解决了响应长度崩溃的问题。我们在数学推理基准和多模态推理场景中进行了大量实验,LUSPO在各种任务中均表现出色。实验证实,LUSPO相比现有方法如GRPO和GSPO代表了一种新颖的、最先进的优化策略。
英文摘要:
Recent applications of Reinforcement Learning with Verifiable Rewards (RLVR) to Large Language Models (LLMs) and Vision-Language Models (VLMs) have demonstrated significant success in enhancing reasoning capabilities for complex tasks. During RLVR training, an increase in response length is often regarded as a key factor contributing to the growth of reasoning ability. However, the patterns of change in response length vary significantly across different RLVR algorithms during the training process. To provide a fundamental explanation for these variations, this paper conducts an in-depth analysis of the components of mainstream RLVR algorithms. We present a theoretical analysis of the factors influencing response length and validate our theory through extensive experimentation. Building upon these theoretical findings, we propose the Length-Unbiased Sequence Policy Optimization (LUSPO) algorithm. Specifically, we rectify the length bias inherent in Group Sequence Policy Optimization (GSPO), rendering its loss function unbiased with respect to response length and thereby resolving the issue of response length collapse. We conduct extensive experiments across mathematical reasoning benchmarks and multimodal reasoning scenarios, where LUSPO consistently achieves superior performance. Empirical results demonstrate that LUSPO represents a novel, state-of-the-art optimization strategy compared to existing methods such as GRPO and GSPO.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。