将坏苹果与好橙子比较：通过联合偏好优化对齐大型语言模型
Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimization
AI总结：
本文提出 JPO，通过在完整指令-响应对上进行联合偏好优化来对齐 LLM，并在摘要与开放式对话任务上分别较 DPO 提升 5.2% 和 3.3% 胜率。
摘要：
一种对齐大型语言模型（LLM）的常用技术依赖于通过比较以固定上下文为条件的多个生成结果来获取人类偏好。然而，这种方法仅依赖于成对比较，即在相同上下文中评估生成结果。尽管对这类条件偏好有效，但它们往往无法涵盖人类偏好细致且多维的本质。在本工作中，我们重新审视传统的偏好获取范式，并提出一个新的轴线：基于对指令-响应对联合引出偏好。不同于为条件排序协议设计的先前偏好优化方法（例如 DPO），我们提出联合偏好优化（Joint Preference Optimization，JPO），这是一种新的偏好优化目标，使被选择的指令-响应对相对于被拒绝的指令-响应对的联合概率获得更高权重。有趣的是，使用 JPO 在联合指令-响应偏好数据上训练的 LLM，在摘要和开放式对话数据集上的胜率分别比使用 DPO 训练的 LLM 高出 5.2% 和 3.3%。我们的发现表明，对指令和响应对的联合偏好可以通过利用更广泛的人类偏好引出谱系，显著增强 LLM 的对齐。数据和代码可在 https://github.com/Hritikbansal/dove 获取。
英文摘要：
A common technique for aligning large language models (LLMs) relies on acquiring human preferences by comparing multiple generations conditioned on a fixed context. This method, however, relies solely on pairwise comparisons, where the generations are evaluated within an identical context. While effective to such conditional preferences often fail to encompass the nuanced and multidimensional nature of human preferences. In this work, we revisit the traditional paradigm of preference acquisition and propose a new axis based on eliciting preferences jointly over the instruction-response pairs. Unlike prior preference optimizations, which are designed for conditional ranking protocols (e.g., DPO), we propose Joint Preference Optimization (JPO), a new preference optimization objective that upweights the joint probability of the chosen instruction-response pair over the rejected instruction-response pair. Interestingly, LLMs trained with joint instruction-response preference data using JPO outperform LLM trained with DPO by $5.2\%$ and $3.3\%$ win-rate for summarization and open-ended dialogue datasets, respectively. Our findings reveal that joint preferences over instruction and response pairs can significantly enhance the alignment of LLMs by tapping into a broader spectrum of human preference elicitation. The data and code is available at https://github.com/Hritikbansal/dove.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
