2D-DPO：利用二维监督扩展直接偏好优化
2D-DPO: Scaling Direct Preference Optimization with 2-Dimensional Supervision
AI总结：
针对 DPO 仅优化标量或一维偏好的局限，本文构建 HelpSteer-2D 数据集并提出 2D-DPO，从片段与方面进行二维偏好优化，实验证明其优于现有方法。
摘要：
近期，直接偏好优化（Direct Preference Optimization，DPO）的进展凭借其简洁性和有效性，显著增强了大型语言模型（LLM）与人类偏好的对齐。然而，现有方法通常优化标量分数或排序奖励，从而忽视了人类偏好的多维性质。在本工作中，我们建议将 DPO 的偏好扩展到两个维度：片段和方面。我们首先引入一个名为 HelpSteer-2D 的二维监督数据集。对于片段维度，我们将响应划分为句子，并为每个片段分配分数。对于方面维度，我们精心设计了若干涵盖响应质量评分准则的标准。借助二维信号作为反馈，我们开发了 2D-DPO 框架，将总体目标分解为多片段和多方面目标。在流行基准上进行的大量实验表明，2D-DPO 的表现优于优化标量或一维偏好的方法。
英文摘要：
Recent advancements in Direct Preference Optimization (DPO) have significantly enhanced the alignment of Large Language Models (LLMs) with human preferences, owing to its simplicity and effectiveness. However, existing methods typically optimize a scalar score or ranking reward, thereby overlooking the multi-dimensional nature of human preferences. In this work, we propose to extend the preference of DPO to two dimensions: segments and aspects. We first introduce a 2D supervision dataset called HelpSteer-2D. For the segment dimension, we divide the response into sentences and assign scores to each segment. For the aspect dimension, we meticulously design several criteria covering the response quality rubrics. With the 2-dimensional signals as feedback, we develop a 2D-DPO framework, decomposing the overall objective into multi-segment and multi-aspect objectives. Extensive experiments on popular benchmarks demonstrate that 2D-DPO performs better than methods that optimize for scalar or 1-dimensional preferences.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
