arXiv 2411.012452024-11-02cs.CL

PMoL: 面向大语言模型对齐偏好混合的参数高效MoE

PMoL: Parameter Efficient MoE for Preference Mixing of LLM Alignment

Harbin Institute of Technology（哈尔滨工业大学）；Beijing Normal University - Hong Kong Baptist University United International College（北京师范大学-香港浸会大学联合国际学院）；Qilu University of Technology（齐鲁工业大学）

Dongxu Liu, Bing Xu, Yinzhuo Chen, Bufan Xu, Wenpeng Lu, Muyun Yang, Tiejun Zhao

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对RLHF难以处理多冲突偏好导致大模型对齐效果下降的问题，提出结合MoE与LoRA的参数高效架构PMoL，通过专家组软损失实现偏好混合，以更低训练成本取得更优的多偏好对齐效果。

摘要：

基于人类反馈的强化学习（RLHF）已被证实是大语言模型（LLM）偏好对齐的有效方法，被广泛应用于LLM的后训练流程中。但RLHF难以处理多个相互冲突的偏好，这会导致LLM与人类偏好的对齐效果下降。为解决该问题，我们从模型架构视角提出了偏好混合低秩适配器（Preference Mixture of LoRAs, PMoL），可适配任意数量的待混合偏好。PMoL结合了专家混合（Mixture of Experts, MoE）与低秩适配器（Low Rank Adaptor, LoRA），该架构被创新性地应用于偏好对齐研究，取得了显著的性能提升。我们采用专家组软损失使MoE具备偏好混合能力。通过奖励模型与GPT-4o的综合评估，实验结果表明，与基线方法相比，PMoL具备更优异的偏好混合能力，能以更低的训练成本实现更好的偏好对齐效果。

英文摘要：

Reinforcement Learning from Human Feedback (RLHF) has been proven to be an effective method for preference alignment of large language models (LLMs) and is widely used in the post-training process of LLMs. However, RLHF struggles with handling multiple competing preferences. This leads to a decrease in the alignment of LLMs with human preferences. To address this issue, we propose Preference Mixture of LoRAs (PMoL) from the perspective of model architecture, which can adapt to any number of preferences to mix. PMoL combines Mixture of Experts (MoE) and Low Rank Adaptor (LoRA). This architecture is innovatively applied to the research of preference alignment and has achieved significant performance improvement. The expert group soft loss is used to enable MoE with the ability to mix preferences. Through comprehensive evaluation by the reward model and GPT-4o, the experiment results show that PMoL has superior preference mixing capabilities compared to baseline methods. PMoL achieves better preference alignment with lower training costs.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。