arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.07832cs.LGcs.AI
收藏

rePIRL: 通过逆强化学习学习PRM以提高LLM推理

rePIRL: Learn PRM with Inverse RL for LLM Reasoning

Meta AI;Department of Computer Science, University of California, Santa Barbara(加州大学圣芭芭拉分校计算机科学系);Google DeepMind(谷歌DeepMind);Independent Researcher(独立研究者)

Xian Wu, Kaijie Zhu, Ying Zhang, Lun Wang, Wenbo Guo

AI总结:

本文提出rePIRL框架,通过逆强化学习学习高效的PRM,无需依赖专家策略的强假设,解决了传统方法中熵崩溃等固有限制问题,通过双学习过程和定制技术提升LLM推理性能,并在数学和编程任务数据集上验证了其有效性。

摘要:

过程奖励已被广泛用于深度强化学习以提高训练效率、减少方差并防止奖励黑客。在LLM推理中,现有工作也探索了各种解决方案来学习有效的过程奖励模型(PRM),有或无专家策略的帮助。然而,现有方法要么依赖于对专家策略的强假设(例如要求其奖励函数),要么受到固有限制(例如熵崩溃),导致PRM效果有限或泛化能力差。在本文中,我们引入了rePIRL,一种受逆强化学习启发的框架,能够在对专家策略假设最少的情况下学习有效的PRM。具体来说,我们设计了一种双学习过程,交替更新策略和PRM。我们的学习算法具有定制技术,以解决将传统逆强化学习扩展到LLM的挑战。我们理论证明,所提出的学习框架可以统一在线和离线PRM学习方法,证明rePIRL可以在最少假设下学习PRM。在标准化数学和编程推理数据集上的经验评估展示了rePIRL在现有方法上的有效性。我们进一步展示了训练的PRM在测试时训练、测试时扩展以及为训练困难问题提供早期信号的应用。最后,我们通过详细的消融研究验证了我们的训练配方和关键设计选择。

英文摘要:

Process rewards have been widely used in deep reinforcement learning to improve training efficiency, reduce variance, and prevent reward hacking. In LLM reasoning, existing works also explore various solutions for learning effective process reward models (PRM) with or without the help of an expert policy. However, existing methods either rely on strong assumptions about the expert policies (e.g., requiring their reward functions) or suffer intrinsic limitations (e.g., entropy collapse), resulting in weak PRMs or limited generalizability. In this paper, we introduce rePIRL, an inverse RL-inspired framework that learns effective PRMs with minimal assumptions about expert policies. Specifically, we design a dual learning process that updates the policy and the PRM interchangeably. Our learning algorithm has customized techniques to address the challenges of scaling traditional inverse RL to LLMs. We theoretically show that our proposed learning framework can unify both online and offline PRM learning methods, justifying that rePIRL can learn PRMs with minimal assumptions. Empirical evaluations on standardized math and coding reasoning datasets demonstrate the effectiveness of rePIRL over existing methods. We further show the application of our trained PRM in test-time training, test-time scaling, and providing an early signal for training hard problems. Finally, we validate our training recipe and key design choices via a detailed ablation study.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑