MidPO: Dual Preference Optimization for Safety and Helpfulness in Large Language Models via a Mixture of Experts Framework
Sun Yat-sen University(中山大学);Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(深圳先进技术研究院、中国科学院);Beijing Normal University(北京师范大学);Central University of Finance and Economics(中央财经大学)
Yupeng Qi, Ziyu Lyu, Min Yang, Yanlin Wang, Lu Bai, Lixin Cui
随着大语言模型(LLMs)在各个领域的应用日益广泛,在保持LLMs有用性的同时增强安全性已成为一项关键挑战。近期研究通过安全约束的在线偏好优化或安全约束的离线偏好优化来解决这一问题。然而,安全约束的在线方法往往存在过度安全的问题,可能降低有用性;而安全约束的离线方法在自适应平衡安全性与有用性方面表现不佳。为解决这些局限性,我们提出了MidPO,一个用于安全性-有用性双重偏好优化的专家混合(Mixture of Experts,MoE)框架。首先,MidPO设计了单偏好增强的直接偏好优化方法,将基础模型转化为两个独立的专家,分别称为安全性专家和有用性专家,并对这两个独立专家进行微调,以实现最优的安全性或有有用性性能。其次,为了在安全性与有用性之间实现有效平衡,MidPO将这两个专家纳入MoE框架,并设计了一种动态路由机制,以自适应地分配每个专家的贡献。我们在三个常用数据集上进行了定量和定性实验,证明所提出的MidPO在安全性和有用性方面均显著优于最先进的方法。代码和模型将公开发布。
英文摘要:
As large language models (LLMs) are increasingly applied across various domains, enhancing safety while maintaining the helpfulness of LLMs has become a critical challenge. Recent studies solve this problem through safety-constrained online preference optimization or safety-constrained offline preference optimization. However, the safety-constrained online methods often suffer from excessive safety, which might reduce helpfulness, while the safety-constrained offline methods perform poorly in adaptively balancing safety and helpfulness. To address these limitations, we propose MidPO, a \textbf{\underline{Mi}}xture of Experts (MoE) framework for safety-helpfulness \textbf{\underline{d}}ual \textbf{\underline{P}}reference \textbf{\underline{O}}ptimization. Firstly, MidPO devises single-preference enhanced direct preference optimization approach to transform the base model into two independent experts, termed safety and helpfulness experts, and fine-tunes the two independent experts for optimal safety or helpfulness performance. Secondly, to achieve an effective balance between safety and helpfulness, MidPO incorporates the two experts into the MoE framework and designs a dynamic routing mechanism to allocate contributions from each expert adaptively. We conduct quantitative and qualitative experiments on three popular datasets to demonstrate the proposed MidPO significantly outperforms state-of-the-art approaches in both safety and helpfulness. The code and models will be released.
Aligning Language Models Using Follow-up Likelihood as Reward Signal
National University of Singapore(新加坡国立大学);Peking University(北京大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Tencent AI Lab(腾讯AI实验室);Shenzhen Research Institute of Big Data(深圳大数据研究院);University of Science and Technology of China(中国科学技术大学)
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);Tencent Inc.(腾讯公司)
Can Vision-Language Models Stay Helpful When Facing Implicit Risks? Intent-Privilege OPSD for Efficient Safety-Helpfulness Alignment
Southern University of Science and Technology(南方科技大学);Sun Yat-sen University(中山大学);Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(广东省人工智能与数字经济实验室(深圳));University of Electronic Science and Technology of China(电子科技大学);Hangzhou Dianzi University(杭州电子科技大学);Shenzhen MSU-BIT University(深圳北理莫斯科大学);GigaAI(极佳科技);The Chinese University of Hong Kong (Shenzhen)(香港中文大学(深圳))
One Model for All: Multi-Objective Controllable Language Models
Ruhr University Bochum(波鸿鲁尔大学);Eindhoven University of Technology(埃因霍温理工大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);University of Liverpool(利物浦大学)
OrthAlign: Orthogonal Subspace Decomposition for Non-Interfering Multi-Objective Alignment
Institute of Artificial Intelligence (TeleAI), China Telecom(人工智能研究院(TeleAI),中国电信);RUC(中国人民大学);USTC(University of Science and Technology of China);NTU(National University of Technology);NUS(National University of Singapore);USC(University of Southern California);SCU(Sichuan University);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Rubrics as an Attack Surface: Stealthy Preference Drift in LLM Judges
University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);Carnegie Mellon University(卡内基梅隆大学);Yale University(耶鲁大学);The University of Texas at Austin(德克萨斯大学奥斯汀分校)
Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models
Key Lab of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (CAS)(智能信息处理重点实验室,计算技术研究所,中国科学院);State Key Lab of AI Safety(人工智能安全国家重点实验室);University of Chinese Academy of Sciences, CAS(中国科学院大学);Zhejiang University(浙江大学)
AI Alignment through a Game-theoretic Lens: A Survey
James Cook University(詹姆斯库克大学);Western Sydney University(西悉尼大学);Adelaide University(阿德莱德大学);CSIRO(联邦科学与工业研究组织);The University of Osaka(大阪大学);Nanjing University of Science and Technology(南京理工大学);Macquarie University(麦考瑞大学);La Trobe University(拉筹伯大学)
SAFER: Advancing Safety Alignment via Efficient Ex-Ante Reasoning
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);ZJU-Hangzhou Global Scientific and Technological Innovation Center, Zhejiang University(浙江大学Hangzhou全球科技创新中心);Polytechnic Institute, Zhejiang University(浙江大学 polytechnic 院);School of Software Technology, Zhejiang University(浙江大学软件技术学院);ZJU-UIUC Institute, Zhejiang University(浙江大学UIUC研究院)