arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.16216cs.CL
收藏

为大型语言模型进行后训练的强化学习:综述

Reinforcement Learning for LLM Post-Training: A Survey

Salesforce;AWS AI Labs;Airbnb

Zhichao Wang, Kiran Ramnath, Bin Bi, Shiva Kumar Pentyala, Sougata Chaudhuri, Shubham Mehrotra, Zixu, Zhu, Xiang-Bo Mao, Sitaram Asur, Na, Cheng

AI总结:

本文综述了通过强化学习进行大型语言模型后训练的方法,分析了RLHF和RLVR等技术,并提出统一的策略梯度框架,旨在为研究人员提供技术参考。

摘要:

大型语言模型(LLMs)通过预训练和监督微调(SFT)训练后,仍可能产生有害或不一致的输出,或在数学和编程等领域表现不佳。基于强化学习(RL)的后训练方法,如通过人类反馈的强化学习(RLHF)方法(如直接偏好优化(DPO)和可验证奖励的强化学习(RLVR)方法(如PPO和GRPO)等,已显著缓解了这些问题。然而,现有研究未对推动这些进展的各种方法进行技术细节的比较。为填补这一空白,本文提出了一项及时的综述,将基础组件与最新进展联系起来。我们推导出一个统一的策略梯度框架,将预训练、SFT、RLHF和RLVR作为特殊案例,并组织其中更近期的技术。本文的主要贡献包括:(1)对MLE、RLHF和RLVR基础以及统一策略梯度框架的自包含介绍;(2)详细分析PPO和GRPO方法以及离线和迭代DPO方法,沿提示采样、响应采样和梯度系数轴分解;(3)标准化符号,实现直接跨方法比较;(4)附录中对每种方法的实现细节和实证结果的全面比较。我们旨在为从事LLM后训练研究的研究人员和实践者提供技术参考。

英文摘要:

Large language models (LLMs) trained via pretraining and supervised fine-tuning (SFT) can still produce harmful and misaligned outputs, or struggle in domains like math and coding. Reinforcement learning (RL)-based post-training methods, including Reinforcement Learning from Human Feedback (RLHF) methods like Direct Preference Optimization (DPO) and Reinforcement Learning with Verifiable Rewards (RLVR) approaches like PPO and GRPO, have made remarkable gains to alleviate these issues. Yet, no existing work offers a technically detailed comparison of the various methods driving this progress. In order to fill this gap, we present a timely survey that connects foundational components with latest advancements. We derive a single policy gradient framework that unifies pretraining, SFT, RLHF, and RLVR as special cases while also organizing the more recent techniques therein. The main contributions of our survey are as follows: (1) a self-contained introduction to MLE, RLHF, and RLVR foundations and the unified policy gradient framework; (2) detailed technical analysis of PPO- and GRPO-based methods alongside offline and iterative DPO approaches, decomposed along prompt sampling, response sampling, and gradient coefficient axes; (3) standardized notation enabling direct cross-method comparison; and (4) comprehensive comparison of implementation details and empirical results of each method in the appendix. We aim to serve as a technically grounded reference for researchers and practitioners working on LLM post-training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑