arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2506.02460cs.CL
收藏

MidPO:通过专家混合框架实现大语言模型安全性与有用性的双重偏好优化

MidPO: Dual Preference Optimization for Safety and Helpfulness in Large Language Models via a Mixture of Experts Framework

Sun Yat-sen University(中山大学);Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(深圳先进技术研究院、中国科学院);Beijing Normal University(北京师范大学);Central University of Finance and Economics(中央财经大学)

Yupeng Qi, Ziyu Lyu, Min Yang, Yanlin Wang, Lu Bai, Lixin Cui

AI总结:

本文提出MidPO框架,通过专家混合与动态路由机制分别优化安全性和有用性专家,实现LLMs安全性与有用性的自适应平衡,并在多个数据集上显著超越现有方法。

摘要:

随着大语言模型(LLMs)在各个领域的应用日益广泛,在保持LLMs有用性的同时增强安全性已成为一项关键挑战。近期研究通过安全约束的在线偏好优化或安全约束的离线偏好优化来解决这一问题。然而,安全约束的在线方法往往存在过度安全的问题,可能降低有用性;而安全约束的离线方法在自适应平衡安全性与有用性方面表现不佳。为解决这些局限性,我们提出了MidPO,一个用于安全性-有用性双重偏好优化的专家混合(Mixture of Experts,MoE)框架。首先,MidPO设计了单偏好增强的直接偏好优化方法,将基础模型转化为两个独立的专家,分别称为安全性专家和有用性专家,并对这两个独立专家进行微调,以实现最优的安全性或有有用性性能。其次,为了在安全性与有用性之间实现有效平衡,MidPO将这两个专家纳入MoE框架,并设计了一种动态路由机制,以自适应地分配每个专家的贡献。我们在三个常用数据集上进行了定量和定性实验,证明所提出的MidPO在安全性和有用性方面均显著优于最先进的方法。代码和模型将公开发布。

英文摘要:

As large language models (LLMs) are increasingly applied across various domains, enhancing safety while maintaining the helpfulness of LLMs has become a critical challenge. Recent studies solve this problem through safety-constrained online preference optimization or safety-constrained offline preference optimization. However, the safety-constrained online methods often suffer from excessive safety, which might reduce helpfulness, while the safety-constrained offline methods perform poorly in adaptively balancing safety and helpfulness. To address these limitations, we propose MidPO, a \textbf{\underline{Mi}}xture of Experts (MoE) framework for safety-helpfulness \textbf{\underline{d}}ual \textbf{\underline{P}}reference \textbf{\underline{O}}ptimization. Firstly, MidPO devises single-preference enhanced direct preference optimization approach to transform the base model into two independent experts, termed safety and helpfulness experts, and fine-tunes the two independent experts for optimal safety or helpfulness performance. Secondly, to achieve an effective balance between safety and helpfulness, MidPO incorporates the two experts into the MoE framework and designs a dynamic routing mechanism to allocate contributions from each expert adaptively. We conduct quantitative and qualitative experiments on three popular datasets to demonstrate the proposed MidPO significantly outperforms state-of-the-art approaches in both safety and helpfulness. The code and models will be released.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑