arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.10958cs.CLcs.AIcs.LG
收藏

相对偏好优化:通过对比相同和多样提示中的响应增强大语言模型对齐

Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts

The University of Texas at Austin(德克萨斯大学奥斯汀分校);Microsoft Azure AI(微软Azure人工智能);Google(谷歌)

Yueqin Yin, Zhendong Wang, Yi Gu, Hai Huang, Weizhu Chen, Mingyuan Zhou

AI总结:

针对直接偏好优化未充分利用相似提示对比信息的问题,提出相对偏好优化,通过对比加权机制利用成对和未成对偏好数据,提升大语言模型与用户偏好对齐及适应性。

摘要:

在大语言模型(LLMs)领域,将模型与用户的多样化偏好对齐是一个关键挑战。直接偏好优化(DPO)在该领域发挥了关键作用。它通过使用源自相同提示的偏好对来工作,并且无需额外的奖励模型即可运行。然而,DPO并未完全反映人类学习的复杂性,人类学习通常涉及不仅对相同问题而且对相似问题的对比响应的理解。为了克服这一不足,我们提出了相对偏好优化(RPO)。RPO旨在区分源自相同和相关提示的更受偏好和较少受偏好的响应。它引入了一种对比加权机制,使得能够使用更广泛的偏好数据(包括成对和未成对集合)来调整大语言模型。这种方法扩展了模型的学习能力,使其能够利用来自更多样化提示集的见解。通过实证测试,包括对话和摘要任务,以及使用AlpacaEval2.0排行榜的评估,RPO展示了在使大语言模型与用户偏好对齐以及提高其在训练过程中的适应性方面的卓越能力。我们的代码可在https://github.com/yinyueqin/relative-preference-optimization查看。

英文摘要:

In the field of large language models (LLMs), aligning models with the diverse preferences of users is a critical challenge. Direct Preference Optimization (DPO) has played a key role in this area. It works by using pairs of preferences derived from the same prompts, and it functions without needing an additional reward model. However, DPO does not fully reflect the complex nature of human learning, which often involves understanding contrasting responses to not only identical but also similar questions. To overcome this shortfall, we propose Relative Preference Optimization (RPO). RPO is designed to discern between more and less preferred responses derived from both identical and related prompts. It introduces a contrastive weighting mechanism, enabling the tuning of LLMs using a broader range of preference data, including both paired and unpaired sets. This approach expands the learning capabilities of the model, allowing it to leverage insights from a more varied set of prompts. Through empirical tests, including dialogue and summarization tasks, and evaluations using the AlpacaEval2.0 leaderboard, RPO has demonstrated a superior ability to align LLMs with user preferences and to improve their adaptability during the training process. Our code can be viewed at https://github.com/yinyueqin/relative-preference-optimization

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑