arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2511.04800cs.CL
收藏

探索强化学习推理语言模型中遗留的数据

Explore Data Left Behind in Reinforcement Learning for Reasoning Language Models

University of Maryland, College Park(马里兰大学学院市分校);ByteDance(字节跳动);Duke University(杜克大学);Pennsylvania State University(宾夕法尼亚州立大学)

Chenxi Liu, Junjie Liang, Yuqi Jia, Bochuan Cao, Yang Bai, Heng Huang, Xun Chen

AI总结:

针对RLVR训练中奖励方差为零的残差提示无法提供训练信号的问题,本文提出ERPO框架,通过历史追踪和自适应提高采样温度来鼓励探索残差提示,重新激活训练信号,在Qwen2.5系列多个数学推理基准上持续超越强基线。

摘要:

带可验证奖励的强化学习(RLVR)已成为提升大语言模型(LLMs)推理能力的有效方法。Group Relative Policy Optimization(GRPO)系列在利用RLVR训练LLMs方面展现出强大性能。然而,随着模型训练时间延长和规模扩大,越来越多的训练提示变为残差提示,即奖励方差为零、无法提供训练信号的提示。因此,贡献于训练的提示数量减少,降低了多样性并阻碍了训练效果。为充分利用这些残差提示,我们提出了Explore Residual Prompts in Policy Optimization(ERPO)框架,该框架鼓励对残差提示进行探索并重新激活其训练信号。ERPO为每个提示维护一个历史追踪器,并对先前产生全部正确响应的残差提示自适应地提高采样温度。这促使模型生成更多样化的推理轨迹,引入错误响应从而恢复训练信号。在Qwen2.5系列上的实验结果表明,ERPO在多个数学推理基准上持续超越强基线方法。

英文摘要:

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an effective approach for improving the reasoning abilities of large language models (LLMs). The Group Relative Policy Optimization (GRPO) family has demonstrated strong performance in training LLMs with RLVR. However, as models train longer and scale larger, more training prompts become residual prompts, those with zero variance rewards that provide no training signal. Consequently, fewer prompts contribute to training, reducing diversity and hindering effectiveness. To fully exploit these residual prompts, we propose the Explore Residual Prompts in Policy Optimization (ERPO) framework, which encourages exploration on residual prompts and reactivates their training signals. ERPO maintains a history tracker for each prompt and adaptively increases the sampling temperature for residual prompts that previously produced all correct responses. This encourages the model to generate more diverse reasoning traces, introducing incorrect responses that revive training signals. Empirical results on the Qwen2.5 series demonstrate that ERPO consistently surpasses strong baselines across multiple mathematical reasoning benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑