State-of-the-art large language models (LLMs) have become indispensable tools for various tasks. However, training LLMs to serve as effective assistants for humans requires careful consideration. A promising approach is reinforcement learning from human feedback (RLHF), which leverages human feedback to update the model in accordance with human preferences and mitigate issues like toxicity and hallucinations. Yet, an understanding of RLHF for LLMs is largely entangled with initial design choices that popularized the method and current research focuses on augmenting those choices rather than fundamentally improving the framework. In this paper, we analyze RLHF through the lens of reinforcement learning principles to develop an understanding of its fundamentals, dedicating substantial focus to the core component of RLHF -- the reward model. Our study investigates modeling choices, caveats of function approximation, and their implications on RLHF training algorithms, highlighting the underlying assumptions made about the expressivity of reward. Our analysis improves the understanding of the role of reward models and methods for their training, concurrently revealing limitations of the current methodology. We characterize these limitations, including incorrect generalization, model misspecification, and the sparsity of feedback, along with their impact on the performance of a language model. The discussion and analysis are substantiated by a categorical review of current literature, serving as a reference for researchers and practitioners to understand the challenges of RLHF and build upon existing efforts.
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
MIT(麻省理工学院);Harvard University(哈佛大学);Columbia University(哥伦比亚大学);Cornell Tech(康奈尔科技校区);Apollo Research(阿波罗研究院);ETH Zurich(苏黎世联邦理工学院);UC Berkeley(加州大学伯克利分校);University of Sussex(萨塞克斯大学);EffiSciences;University of Cambridge(剑桥大学);New York University(纽约大学);UNC Chapel Hill(北卡罗来纳大学教堂山分校);University of Southern California(南加州大学);Stanford University(斯坦福大学)
ACE-RLHF: Automated Code Evaluation and Socratic Feedback Generation Tool using Large Language Models and Reinforcement Learning with Human Feedback
Department of Computer Science, Cleveland State University(克利夫兰州立大学计算机科学系);School of Computing and Information Sciences, Florida International University(佛罗里达国际大学计算与信息科学学院);Data Science and Learning Division, Argonne National Laboratory(阿贡国家实验室数据科学与学习分部)
UCL Centre for Artificial Intelligence(伦敦大学学院人工智能中心);Department of Computer Science(计算机科学系);University College London(伦敦大学学院);UK AI Security Institute(英国人工智能安全研究所);Department of Computer Science and Engineering(计算机科学与工程系);University of Bologna(博洛尼亚大学)
Reinforcement Learning from Human Feedback: A Statistical Perspective
Department of Biostatistics, Yale University(耶鲁大学生物统计学系);Department of Statistics, London School of Economics and Political Science(伦敦政治经济学院统计学系);Mitch Daniels School of Business, Purdue University(普渡大学米奇·丹尼尔斯商学院)
Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Salesforce AI Research(Salesforce AI研究院);The Hong Kong University of Science and Technology(香港科技大学)
Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles
National University of Defense Technology(国防科技大学);State Key Laboratory of Complex & Critical Software Environment(复杂与关键软件环境国家重点实验室);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Peng Cheng Laboratory(鹏城实验室)
RLHF Fine-Tuning of LLMs for Alignment with Implicit User Feedback in Conversational Recommenders
Northeastern University(东北大学);Washington University in St. Louis(圣路易斯华盛顿大学);Northwestern University(西北大学);University of Nebraska - Lincoln(内布拉斯加大学林肯分校);Columbia University(哥伦比亚大学)
Low-Rank Contextual Reinforcement Learning from Heterogeneous Human Feedback
Department of Statistics and Operations Research, University of North Carolina, Chapel Hill(统计与运筹学系,北卡罗来纳大学 Chapel Hill 分校);Daniels School of Business, Purdue University(商务学院,普渡大学);Department of Statistics and Operations Research, Department of Genetics, Department of Biostatistics, The University of North Carolina at Chapel Hill(统计与运筹学系,遗传学系,生物统计学系,北卡罗来纳大学 Chapel Hill 分校)
Proxy-RLHF: Decoupling Generation and Alignment in Large Language Model with Proxy
University of Science and Technology of China(中国科学技术大学);Deep Space Exploration Laboratory(深空探测实验室);Institute for Advanced Algorithms Research(先进算法研究院);Sangfor Technologies Inc.(深信服科技股份有限公司);Research Institute of China Telecom(中国电信研究院)