arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2309.06256cs.LG
收藏

缓解RLHF的对齐税

Mitigating the Alignment Tax of RLHF

Princeton University(普林斯顿大学);The Hong Kong University of Science and Technology(香港科技大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);NVIDIA(英伟达)
Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jianmeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hanning Zhang, Hanze Dong, Renjie Pi, Han Zhao, Nan… 展开作者

Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jianmeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hanning Zhang, Hanze Dong, Renjie Pi, Han Zhao, Nan Jiang, Heng Ji, Yuan Yao, Tong Zhang

AI总结:

本文研究RLHF的对齐税问题,发现模型平均能实现最优对齐-遗忘Pareto前沿,并提出异质模型平均(HMA)方法,通过为不同层寻找组合比例,在最大化对齐性能的同时最小化对齐税。

摘要:

LLMs在预训练期间获得了广泛的能力,但在Reinforcement Learning with Human Feedback (RLHF)下对齐LLMs可能导致遗忘预训练能力,这也被称为对齐税。为了研究对齐税,我们使用OpenLLaMA-3B对现有的RLHF算法进行了实验,揭示了在NLP任务中显著的对齐税。然而,尽管有各种缓解遗忘的技术,它们通常与RLHF性能相冲突,导致对齐性能与遗忘缓解之间的权衡,进而形成对齐-遗忘权衡。在本文中,我们展示了模型平均(简单地在RLHF前后模型权重之间进行插值)令人惊讶地在广泛的竞争方法中实现了最强的对齐-遗忘Pareto前沿。为了理解其有效性,我们为模型平均提供了理论见解,揭示它通过在任务共享重叠特征空间的层上增加特征多样性来增强性能Pareto前沿。经验证据通过展示平均低级transformer层的好处证实了我们的分析。基于分析和观察到平均transformer的不同层会导致显著不同的对齐-遗忘权衡,我们提出了Heterogeneous Model Averaging (HMA)来异质地寻找模型层的各种组合比例。HMA旨在最大化对齐性能同时产生最小的对齐税。此外,我们在OpenLLaMA-3B上的一系列RLHF算法中验证了HMA的性能,并进一步将我们的发现扩展到Mistral-7B,该模型由开源偏好模型和GPT4评估。代码可在此处获取:https://github.com/avalonstrel/Mitigating-the-Alignment-Tax-of-RLHF.git。

英文摘要:

LLMs acquire a wide range of abilities during pre-training, but aligning LLMs under Reinforcement Learning with Human Feedback (RLHF) can lead to forgetting pretrained abilities, which is also known as the alignment tax. To investigate alignment tax, we conducted experiments with existing RLHF algorithms using OpenLLaMA-3B, which revealed a pronounced alignment tax in NLP tasks. Whereas, despite various techniques to mitigate forgetting, they are often at odds with the RLHF performance, leading to a trade-off between alignment performance and forgetting mitigation, leading to an alignment-forgetting trade-off. In this paper we show that model averaging, which simply interpolates between pre and post RLHF model weights, surprisingly achieves the most strongest alignment-forgetting Pareto front among a wide range of competing methods. To understand its effectiveness, we offer theoretical insights into model averaging, revealing that it enhances performance Pareto front by increasing feature diversity on the layers where tasks share overlapped feature spaces. Empirical evidence corroborates our analysis by showing the benefits of averaging low-level transformer layers. Building on the analysis and the observation that averaging different layers of the transformer leads to significantly different alignment-forgetting trade-offs, we propose Heterogeneous Model Averaging (HMA) to Heterogeneously find various combination ratios of model layers. HMA seeks to maximize the alignment performance while incurring minimal alignment tax. Moreover, we validate HMA's performance across a range of RLHF algorithms over OpenLLaMA-3B and further extend our findings to Mistral-7B which is evaluated by open-sourced preference model and GPT4. Code available here: https://github.com/avalonstrel/Mitigating-the-Alignment-Tax-of-RLHF.git.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑