本研究首次公开复现了OpenAI开创性TL;DR摘要工作中报告的Reinforcement Learning from Human Feedback (RLHF)缩放行为。我们从零构建RLHF流水线,列举超过20个关键实现细节,并分享复现过程中的核心见解。经RLHF训练的Pythia模型在响应质量上展现出随模型规模扩展的显著提升,其中2.8B和6.9B模型优于OpenAI发布的1.3B checkpoint。我们公开发布训练好的模型checkpoints和代码,以促进后续研究并加速该领域进展。
英文摘要:
This work is the first to openly reproduce the Reinforcement Learning from Human Feedback (RLHF) scaling behaviors reported in OpenAI's seminal TL;DR summarization work. We create an RLHF pipeline from scratch, enumerate over 20 key implementation details, and share key insights during the reproduction. Our RLHF-trained Pythia models demonstrate significant gains in response quality that scale with model size, with our 2.8B, 6.9B models outperforming OpenAI's released 1.3B checkpoint. We publicly release the trained model checkpoints and code to facilitate further research and accelerate progress in the field (\url{https://github.com/vwxyzjn/summarize_from_feedback_details}).
KL-Regularised Q-Learning: A Token-level Action-Value perspective on Online RLHF
Computational and Biological Learning Group, Department of Engineering, University of Cambridge, Cambridge, UK(计算生物学学习组,工程系,剑桥大学,剑桥,英国);Department of Computer Science and Technology, University of Cambridge, Cambridge, UK(计算机科学与技术系,剑桥大学,剑桥,英国);Statistics Laboratory, Department of Pure Mathematics and Mathematical Statistics, University of Cambridge, UK(统计实验室,纯粹数学与数学统计系,剑桥大学,英国)
ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models
School of Data Science, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)数据科学学院);Shenzhen Research Institute of Big Data(深圳大数据研究院);National Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);Pazhou Laboratory (Huangpu)(琶洲实验室(黄埔));China Shenzhen International Center For Industrial and Applied Mathematics(中国深圳国际工业与应用数学中心)
Reveal the Mystery of DPO: The Connection between DPO and RL Algorithms
School of Mathematics and Statistics(数学与统计学院);Beijing Jiaotong University(北京交通大学);University of Science and Technology of China(中国科学技术大学);School of information(信息学院);School of Management(管理学院);Renmin University of China(中国人民大学);Fudan University(复旦大学);Academy of Mathematics and Systems Science(数学与系统科学研究院)
RLHF Fine-Tuning of LLMs for Alignment with Implicit User Feedback in Conversational Recommenders
Northeastern University(东北大学);Washington University in St. Louis(圣路易斯华盛顿大学);Northwestern University(西北大学);University of Nebraska - Lincoln(内布拉斯加大学林肯分校);Columbia University(哥伦比亚大学)
Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Salesforce AI Research(Salesforce AI研究院);The Hong Kong University of Science and Technology(香港科技大学)
RLHF in an SFT Way: From Optimal Solution to Reward-Weighted Alignment
Shenzhen Research Institute of Big Data(深圳大数据研究院);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Qwen LLM Application Team, Alibaba(阿里巴巴Qwen大模型应用团队);Stanford University(斯坦福大学)
Symmetric Reinforcement Learning Loss for Robust Learning on Diverse Tasks and Model Scales
Department of Computer Science and Engineering, The Ohio State University(计算机科学与工程系,俄亥俄州立大学);Department of Computer Science(计算机科学系);Engineering, The Ohio State University(工程系,俄亥俄州立大学)