arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2507.04136cs.AI
收藏

大语言模型强化学习技术的技术综述

A Technical Survey of Reinforcement Learning Techniques for Large Language Models

University of Georgia(佐治亚大学);Purdue University(普渡大学)

Saksham Sahai Srivastava, Vaneet Aggarwal

AI总结:

综述强化学习与语言模型整合,介绍如近端策略优化等算法,分析其在各领域应用,对失败模式算法分析,给出分类法,评估趋势并探讨挑战与新兴方向,为研究提供路线图。

摘要:

本综述为强化学习与语言模型的整合提供了全面基础,突出近端策略优化等算法,对针对大语言模型的强化学习技术进行技术概述,系统分析其应用,对失败模式进行算法分析,给出比较分类法,评估趋势,探讨挑战与新兴方向,为推进基于强化学习的大语言模型发展的研究人员提供路线图。

英文摘要:

This survey offers a comprehensive foundation on the integration of RL with language models, highlighting prominent algorithms such as Proximal Policy Optimization (PPO), Q-Learning, and Actor-Critic methods. Additionally, it provides an extensive technical overview of RL techniques specifically tailored for LLMs, including foundational methods like Reinforcement Learning from Human Feedback (RLHF) and AI Feedback (RLAIF), as well as advanced strategies such as Direct Preference Optimization (DPO) and Group Relative Policy Optimization (GRPO). We systematically analyze their applications across domains, i.e., from code generation to tool-augmented reasoning. Crucially, we move beyond descriptive categorization to provide a rigorous algorithmic analysis of failure modes, mathematically framing the structural bottlenecks and stability trade-offs inherent in policy optimization. We also present a comparative taxonomy based on reward modeling, feedback mechanisms, and optimization strategies. Our evaluation highlights key trends. RLHF remains dominant for alignment, and outcome-based RL such as Reinforcement Learning with Verifiable Rewards (RLVR) significantly improves stepwise reasoning. However, persistent challenges such as reward hacking, computational costs, and scalable feedback collection underscore the need for continued innovation. We also explicate the causal factors behind recent benchmark performances, distinguishing between gains derived from architectural scaling versus those stemming from specific optimization objectives. We further discuss emerging directions, including hybrid RL algorithms, verifier-guided training, and multi-objective alignment frameworks. This survey serves as a roadmap for researchers advancing RL-driven LLM development, balancing capability enhancement with safety and scalability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑