Princeton University(普林斯顿大学);The Hong Kong University of Science and Technology(香港科技大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);NVIDIA(英伟达)
LLMs在预训练期间获得了广泛的能力,但在Reinforcement Learning with Human Feedback (RLHF)下对齐LLMs可能导致遗忘预训练能力,这也被称为对齐税。为了研究对齐税,我们使用OpenLLaMA-3B对现有的RLHF算法进行了实验,揭示了在NLP任务中显著的对齐税。然而,尽管有各种缓解遗忘的技术,它们通常与RLHF性能相冲突,导致对齐性能与遗忘缓解之间的权衡,进而形成对齐-遗忘权衡。在本文中,我们展示了模型平均(简单地在RLHF前后模型权重之间进行插值)令人惊讶地在广泛的竞争方法中实现了最强的对齐-遗忘Pareto前沿。为了理解其有效性,我们为模型平均提供了理论见解,揭示它通过在任务共享重叠特征空间的层上增加特征多样性来增强性能Pareto前沿。经验证据通过展示平均低级transformer层的好处证实了我们的分析。基于分析和观察到平均transformer的不同层会导致显著不同的对齐-遗忘权衡,我们提出了Heterogeneous Model Averaging (HMA)来异质地寻找模型层的各种组合比例。HMA旨在最大化对齐性能同时产生最小的对齐税。此外,我们在OpenLLaMA-3B上的一系列RLHF算法中验证了HMA的性能,并进一步将我们的发现扩展到Mistral-7B,该模型由开源偏好模型和GPT4评估。代码可在此处获取:https://github.com/avalonstrel/Mitigating-the-Alignment-Tax-of-RLHF.git。
英文摘要:
LLMs acquire a wide range of abilities during pre-training, but aligning LLMs under Reinforcement Learning with Human Feedback (RLHF) can lead to forgetting pretrained abilities, which is also known as the alignment tax. To investigate alignment tax, we conducted experiments with existing RLHF algorithms using OpenLLaMA-3B, which revealed a pronounced alignment tax in NLP tasks. Whereas, despite various techniques to mitigate forgetting, they are often at odds with the RLHF performance, leading to a trade-off between alignment performance and forgetting mitigation, leading to an alignment-forgetting trade-off.
In this paper we show that model averaging, which simply interpolates between pre and post RLHF model weights, surprisingly achieves the most strongest alignment-forgetting Pareto front among a wide range of competing methods. To understand its effectiveness, we offer theoretical insights into model averaging, revealing that it enhances performance Pareto front by increasing feature diversity on the layers where tasks share overlapped feature spaces. Empirical evidence corroborates our analysis by showing the benefits of averaging low-level transformer layers. Building on the analysis and the observation that averaging different layers of the transformer leads to significantly different alignment-forgetting trade-offs, we propose Heterogeneous Model Averaging (HMA) to Heterogeneously find various combination ratios of model layers. HMA seeks to maximize the alignment performance while incurring minimal alignment tax. Moreover, we validate HMA's performance across a range of RLHF algorithms over OpenLLaMA-3B and further extend our findings to Mistral-7B which is evaluated by open-sourced preference model and GPT4. Code available here: https://github.com/avalonstrel/Mitigating-the-Alignment-Tax-of-RLHF.git.
Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles
National University of Defense Technology(国防科技大学);State Key Laboratory of Complex & Critical Software Environment(复杂与关键软件环境国家重点实验室);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Peng Cheng Laboratory(鹏城实验室)
Towards Reward Fairness in RLHF: From a Resource Allocation Perspective
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学人工智能学院);Beijing Key Laboratory of Research on Large Models and Intelligent Governance(北京大型模型与智能治理研究重点实验室);Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE(下一代智能搜索与推荐工程技术研究中心,教育部);Kuaishou Technology(快手科技);University of Chinese Academy of Sciences(中国科学院大学)
Certifiable Safe RLHF: Semantic Grounding and Fixed Penalty Constraint Optimization for Safer LLM Alignment
Department of Electrical and Computer Engineering(电气与计算机工程系);New Jersey Institute of Technology(新泽西理工学院);Department of Computer Engineering(计算机工程系);Heritage Institute of Technology(遗产理工学院)
Mitigating Reward Over-optimization in Direct Alignment Algorithms with Importance Sampling
VinUniversity(文大学);VinUni-Illinois Smart Health Center(VinUni-伊利诺伊智能健康中心);University of Stuttgart(斯图加特大学);University of California Los Angeles (UCLA)(加州大学洛杉矶分校);International University - VNUHCM(国际大学 - VNUHCM);University of Science - VNUHCM(科学大学 - VNUHCM);German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心);Oldenburg University(奥尔登堡大学);Max Planck Research School for Intelligent Systems (IMPRS-IS)(马克斯·普朗克智能系统研究学校)
Towards Efficient Online Exploration for Reinforcement Learning with Human Feedback
Department of Statistics and Data Science, The Chinese University of Hong Kong(统计与数据科学系,香港中文大学);Department of Statistics, University of Wisconsin-Madison(统计系,威斯康星大学麦迪逊分校)
Reinforcement Learning with Semantic Rewards Enables Low-Resource Language Expansion without Alignment Tax
Minzu University of China(中国民族大学);Ant Group(蚂蚁集团);Shanghai Jiao Tong University(上海交通大学);University of Macau(澳门大学);Peking University(北京大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Hainan International College, Minzu University of China(中国民族大学海南国际学院)
Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularization
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);The State Key Lab of Brain-Machine Intelligence, Zhejiang University(浙江大学脑机智能国家重点实验室);LLM Safety Centre, Beijing Academy of Artificial Intelligence(北京人工智能研究院大语言模型安全中心);Center for AI Safety and Governance, Peking University(北京大学人工智能安全与治理中心)
Tail-Aware Information-Theoretic Bounds for LLM Alignment under Heavy-Tailed Rewards
Institute of Artificial Intelligence, Beihang University(北京航空航天大学人工智能研究院);Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing(北京未来区块链与隐私计算高精尖创新中心);Advanced Institute of Information Technology, Peking University(北京大学信息技术高等研究院);Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机技术研究所);Computer and Mathematical Sciences, Computer Science, and Statistics, University of Toronto(多伦多大学计算机与数学科学、计算机科学和统计学系);MBZUAI(穆罕默德·本·扎耶德人工智能大学)
Safety Alignment as Continual Learning: Mitigating the Alignment Tax via Orthogonal Gradient Projection
School of Life Sciences, IDG/McGovern Institute for Brain Research(生命科学学院,IDG/麦戈文脑科学研究所);Dept. of Comp. Sci. and Tech., Institute for AI, Tsinghua-Bosch Joint ML Center, THBI Lab, BNRist Center(计算机科学与技术系,人工智能研究所,清华大学-博世联合机器学习中心,THBI实验室,BNRist中心);Tsinghua University, Beijing, China(清华大学,北京,中国)