arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2607.01392cs.CL
收藏

基于互信息的多目标探索与偏好优化

Multi-Objective Exploration and Preference Optimization via Mutual Information

School of Computer, Beihang University(北京航空航天大学计算机学院);Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd(星辰AGI实验室,中国电信人工智能技术(北京)有限公司)

Hongyan Xie, Yikun Ban, Ruiyu Fang, Zixuang Huang, Deqing Wang, Jianxin Li, Shuangyong Song

AI总结:

提出MI-EPO框架,通过最大化生成响应、偏好反馈和偏好向量的联合条件互信息,统一多目标探索与对齐,实现可控且稳定的多目标权衡。

摘要:

将大型语言模型与多样且异质的人类价值观对齐需要多目标对齐方法,以有效权衡冲突的偏好维度。当前方法通过训练基于偏好向量的策略并利用在线直接偏好优化来实现这种权衡。然而,探索不确定性可能导致不同偏好向量下生成的响应的奖励分布重叠,且生成的响应可能无法有效对齐相应的偏好向量。在本文中,我们提出基于互信息的多目标探索与偏好优化(MI-EPO),一个信息论框架。它通过最大化生成响应、偏好反馈和偏好向量之间的联合条件互信息,统一了多目标探索与对齐。通过引入概率路由机制,MI-EPO自然地分解了目标对齐和偏好感知探索,鼓励模型生成可区分且与不同偏好条件对齐的响应。在安全对齐和有用助手任务上的实验表明,MI-EPO显著改善了生成响应与偏好向量之间的对齐,使输出更可控,并在多个目标上实现了稳定的权衡。

英文摘要:

Aligning large language models with diverse and heterogeneous human values requires multi-objective alignment methods to effectively trade off conflicting preference dimensions. Current methods achieve this trade-off by training policies conditioned on preference vectors and leveraging online direct preference optimization. However, exploration uncertainty can cause the reward distributions of responses generated under different preference vectors to overlap, and the generated responses may fail to effectively align with the corresponding preference vectors. In this paper, we propose Multi-Objective Exploration and Preference Optimization via Mutual Information (MI-EPO), an information-theoretic framework. It unifies multi-objective exploration and alignment by maximizing the joint conditional mutual information among generated responses, preference feedback, and preference vectors. By incorporating a probabilistic routing mechanism, MI-EPO naturally decomposes objective alignment and preference-aware exploration, encouraging the model to generate responses that are distinguishable and aligned with different preference conditions. Experiments on safe alignment and helpful assistant tasks show that MI-EPO significantly improves the alignment between generated responses and preference vectors, makes the outputs more controllable, and achieves stable trade-offs across multiple objectives.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑