arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.08555cs.LGcs.AIcs.CL
收藏

RLHF 解析:对大语言模型基于人类反馈的强化学习的批判性分析

RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs

Princeton University(普林斯顿大学);Georgia Tech(佐治亚理工学院);University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校)

Shreyas Chaudhari, Pranjal Aggarwal, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, Karthik Narasimhan, Ameet Deshpande, Bruno Castro da Silva

AI总结:

本文从强化学习原理出发批判性分析 LLM 的 RLHF,重点剖析奖励模型的建模假设、训练方法与局限,并分类综述文献以为后续研究提供参考。

摘要:

最先进的大语言模型(LLM)已成为各类任务中不可或缺的工具。然而,训练 LLM 使其成为人类的高效助手需要审慎考量。一种颇具前景的方法是基于人类反馈的强化学习(RLHF),它利用人类反馈按照人类偏好更新模型,并缓解毒性和幻觉等问题。但是,对用于 LLM 的 RLHF 的理解在很大程度上与使该方法普及的初始设计选择纠缠在一起,而当前研究侧重于增强这些选择,而非从根本上改进该框架。在本文中,我们通过强化学习原理的视角分析 RLHF,以理解其基础,并将大量重点放在 RLHF 的核心组成部分——奖励模型上。我们的研究考察了建模选择、函数逼近的注意事项,以及它们对 RLHF 训练算法的影响,强调了关于奖励表达能力所作出的潜在假设。我们的分析增进了对奖励模型作用及其训练方法的理解,同时揭示了当前方法论的局限性。我们刻画了这些局限性,包括错误泛化、模型误设和反馈稀疏性,以及它们对语言模型性能的影响。相关讨论和分析得到了对当前文献进行分类综述的支撑,可作为研究人员和从业者理解 RLHF 挑战并在现有工作基础上推进的参考。

英文摘要:

State-of-the-art large language models (LLMs) have become indispensable tools for various tasks. However, training LLMs to serve as effective assistants for humans requires careful consideration. A promising approach is reinforcement learning from human feedback (RLHF), which leverages human feedback to update the model in accordance with human preferences and mitigate issues like toxicity and hallucinations. Yet, an understanding of RLHF for LLMs is largely entangled with initial design choices that popularized the method and current research focuses on augmenting those choices rather than fundamentally improving the framework. In this paper, we analyze RLHF through the lens of reinforcement learning principles to develop an understanding of its fundamentals, dedicating substantial focus to the core component of RLHF -- the reward model. Our study investigates modeling choices, caveats of function approximation, and their implications on RLHF training algorithms, highlighting the underlying assumptions made about the expressivity of reward. Our analysis improves the understanding of the role of reward models and methods for their training, concurrently revealing limitations of the current methodology. We characterize these limitations, including incorrect generalization, model misspecification, and the sparsity of feedback, along with their impact on the performance of a language model. The discussion and analysis are substantiated by a categorical review of current literature, serving as a reference for researchers and practitioners to understand the challenges of RLHF and build upon existing efforts.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑