ULMA：结合人类演示与逐点偏好的统一语言模型对齐
ULMA: Unified Language Model Alignment with Human Demonstration and Point-wise Preference
AI总结：
本文提出逐点直接偏好优化和统一语言模型对齐方法，将人类演示与逐点偏好统一到单步对齐流程中，并通过实验验证其有效性。
摘要：
将语言模型对齐到人类期望（例如有用且无害）已成为大语言模型面临的紧迫挑战。典型的对齐流程包括监督微调和偏好学习。大多数偏好学习方法（如 RLHF 和 DPO）依赖成对偏好数据，这无法充分应对人类反馈为逐点形式的场景，从而导致潜在的信息损失和次优性能。为弥补这一不足，我们提出了逐点直接偏好优化（Point-wise Direct Preference Optimization），这是一种旨在有效利用逐点反馈的新型偏好学习方法。我们的工作还揭示了监督微调与逐点偏好学习之间的新联系，最终形成了统一语言模型对齐（Unified Language Model Alignment），这是一种将人类演示对齐与逐点偏好对齐统一起来的单步方法。在具有二值或连续标签的逐点偏好数据集上的大量实验验证了我们方法的有效性。我们发布了代码以及一个包含高质量无害性演示样本的新数据集。
英文摘要：
Aligning language models to human expectations, e.g., being helpful and harmless, has become a pressing challenge for large language models. A typical alignment procedure consists of supervised fine-tuning and preference learning. Most preference learning methods, such as RLHF and DPO, depend on pairwise preference data, which inadequately address scenarios where human feedback is point-wise, leading to potential information loss and suboptimal performance. Addressing this gap, we introduce Point-wise Direct Preference Optimization, a novel preference learning method designed to harness point-wise feedback effectively. Our work also uncovers a novel connection between supervised fine-tuning and point-wise preference learning, culminating in Unified Language Model Alignment, a single-step method that unifies the alignment with human demonstrations and point-wise preferences. Extensive experiments on point-wise preference datasets with binary or continuous labels validate the effectiveness of our methods. Our code and a new dataset with high-quality demonstration samples on harmlessness are released.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
