arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2511.23391cs.CL
收藏

模糊性意识优化:朝着直接偏好优化的语义消歧

Ambiguity Awareness Optimization: Towards Semantic Disambiguation for Direct Preference Optimization

AI Technology Center of OVB, Tencent, China(腾讯OVB人工智能技术中心,中国);School of Computer Science, Peking University, China(北京大学计算机学院,中国)

Jian Li, Shenglin Yin, Yujia Zhang, Alan Zhao, Xi Chen, Xiaohui Zhou, Pengfei Xu

AI总结:

本文提出模糊性意识优化(AAO)方法,通过计算偏好对的语义相似性自动重新加权模糊内容,有效提升直接偏好优化的性能。

摘要:

直接偏好优化(DPO)是一种在各种领域中广泛使用的基于人类反馈的强化学习(RLHF)方法。最近的研究越来越多地关注令牌重要性在提高DPO有效性中的作用。观察到在偏好对中经常出现相同或语义相似的内容(定义为模糊内容)。我们假设在DPO训练过程中存在模糊内容可能会引入模糊性,从而限制进一步的对齐改进。通过数学分析和概念验证实验,我们揭示了模糊内容可能潜在地引入模糊性,从而降低性能。为了解决这个问题,我们引入了模糊性意识优化(AAO),一种简单而有效的方法,自动重新加权模糊内容以减少模糊性,通过从偏好对中计算语义相似性。通过广泛的实验,我们证明AAO在多个模型规模和广泛采用的基准数据集中,包括AlpacaEval 2、MT-Bench和Arena-Hard,均能显著超越最先进的方法,而不会明显增加响应长度。具体而言,AAO在AlpacaEval 2上比DPO高出多达8.9分,在Arena-Hard上达到最高15.0分的改进。

英文摘要:

Direct Preference Optimization (DPO) is a widely used reinforcement learning from human feedback (RLHF) method across various domains. Recent research has increasingly focused on the role of token importance in improving DPO effectiveness. It is observed that identical or semantically similar content (defined as ambiguous content) frequently appears within the preference pairs. We hypothesize that the presence of ambiguous content during DPO training may introduce ambiguity, thereby limiting further improvements in alignment. Through mathematical analysis and proof-of-concept experiments, we reveal that ambiguous content may potentially introduce ambiguities, thereby degrading performance. To address this issue, we introduce Ambiguity Awareness Optimization (AAO), a simple yet effective approach that automatically re-weights ambiguous content to reduce ambiguities by calculating semantic similarity from preference pairs. Through extensive experiments, we demonstrate that AAO consistently and significantly surpasses state-of-the-art approaches in performance, without markedly increasing response length, across multiple model scales and widely adopted benchmark datasets, including AlpacaEval 2, MT-Bench, and Arena-Hard. Specifically, AAO outperforms DPO by up to 8.9 points on AlpacaEval 2 and achieves an improvement of by up to 15.0 points on Arena-Hard.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑