回收失败:通过细粒度反策略指导在RLVR中恢复探索
Recycling Failures: Salvaging Exploration in RLVR via Fine-Grained Off-Policy Guidance
AI总结:
SCOPE通过细粒度反策略指导在RLVR中恢复探索,提高轨迹多样性13.5%,在数学推理和分布外推理任务中取得新突破。
摘要:
可验证奖励强化学习(RLVR)已逐渐成为增强大型推理模型复杂推理能力的一种强大范式。然而,标准基于结果的监督存在一个关键限制,即对那些大部分正确但因几个错误步骤而失败的轨迹的惩罚程度与完全错误的轨迹相同。这种粗略的反馈信号导致模型丢弃有价值的大部分正确轨迹,从而降低轨迹多样性,使探索空间过早变窄。过程奖励模型在提供测试时间扩展的可靠逐步验证方面已显示出有效性,但简单地将这些信号整合到RLVR中作为密集奖励却效果不佳。先前的方法试图引入反策略引导的完整轨迹替换,但通常超出策略模型的分布范围,但仍未能有效利用模型自身生成的大部分正确轨迹,因此无法有效缓解探索空间的变窄。为了解决这些问题,我们提出了SCOPE(用于策略探索的逐步修正),一种新的框架,利用过程奖励模型来定位子最优轨迹中的第一个错误步骤,并应用细粒度、逐步的反策略修正。通过在部分正确轨迹上进行精确的细化,我们的方法有效回收了部分正确轨迹,并将多样性分数提高了13.5%,从而维持了广大的探索空间。广泛的实验表明,我们的方法建立了新的最先进结果,在数学推理上实现了平均准确率46.6%,并在分布外推理任务中表现出鲁棒的泛化能力,准确率达到53.4%。
英文摘要:
Reinforcement Learning from Verifiable Rewards (RLVR) has emerged as a powerful paradigm for enhancing the complex reasoning capabilities of Large Reasoning Models. However, standard outcome-based supervision suffers from a critical limitation that penalizes trajectories that are largely correct but fail due to several missteps as heavily as completely erroneous ones. This coarse feedback signal causes the model to discard valuable largely correct rollouts, leading to a degradation in rollout diversity that prematurely narrows the exploration space. Process Reward Models have demonstrated efficacy in providing reliable step-wise verification for test-time scaling, naively integrating these signals into RLVR as dense rewards proves ineffective.Prior methods attempt to introduce off-policy guided whole-trajectory replacement that often outside the policy model's distribution, but still fail to utilize the largely correct rollouts generated by the model itself and thus do not effectively mitigate the narrowing of the exploration space. To address these issues, we propose SCOPE (Step-wise Correction for On-Policy Exploration), a novel framework that utilizes Process Reward Models to pinpoint the first erroneous step in suboptimal rollouts and applies fine-grained, step-wise off-policy rectification. By applying precise refinement on partially correct rollout, our method effectively salvages partially correct trajectories and increases diversity score by 13.5%, thereby sustaining a broad exploration space. Extensive experiments demonstrate that our approach establishes new state-of-the-art results, achieving an average accuracy of 46.6% on math reasoning and exhibiting robust generalization with 53.4% accuracy on out-of-distribution reasoning tasks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。