arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2509.04501cs.CLcs.AIcs.LG
收藏

理解用于模型训练的强化学习,以及 GRAPE 的未来方向

Understanding Reinforcement Learning for Model Training, and future directions with GRAPE

Rohit Patel(独立研究者)

Rohit Patel

AI总结:

本文系统梳理并从零推导 LLM 指令微调中的 SFT、PPO、GRPO、DPO 等强化学习算法,综述后续技术,并提出 GRAPE 研究框架。

摘要:

本文对模型指令微调的关键算法进行了自包含、从零开始的阐述,包括 SFT、拒绝采样、REINFORCE、信赖域策略优化(TRPO)、近端策略优化(PPO)、组相对策略优化(GRPO)和直接偏好优化(DPO)。这些算法的现有解释通常预设先验知识、缺少关键细节,和/或过于泛化且复杂。本文使用面向 LLM 的简化且明确的符号,逐步讨论并推导每种方法,旨在消除歧义,并对相关概念形成清晰、直观的理解。通过尽量减少转向更广泛的强化学习文献,并将概念与 LLM 联系起来,我们去除了多余的抽象,降低了认知负担。在上述阐述之后,本文对所列算法之外的新技术和方法进行了文献综述。最后,以 GRAPE(Generalized Relative Advantage Policy Evolution,广义相对优势策略演化)的形式提出了供研究和探索的新思路。

英文摘要:

This paper provides a self-contained, from-scratch, exposition of key algorithms for instruction tuning of models: SFT, Rejection Sampling, REINFORCE, Trust Region Policy Optimization (TRPO), Proximal Policy Optimization (PPO), Group Relative Policy Optimization (GRPO), and Direct Preference Optimization (DPO). Explanations of these algorithms often assume prior knowledge, lack critical details, and/or are overly generalized and complex. Here, each method is discussed and developed step by step using simplified and explicit notation focused on LLMs, aiming to eliminate ambiguity and provide a clear and intuitive understanding of the concepts. By minimizing detours into the broader RL literature and connecting concepts to LLMs, we eliminate superfluous abstractions and reduce cognitive overhead. Following this exposition, we provide a literature review of new techniques and approaches beyond those detailed. Finally, new ideas for research and exploration in the form of GRAPE (Generalized Relative Advantage Policy Evolution) are presented.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑