Cal-DPO：用于语言模型对齐的校准直接偏好优化
Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
AI总结：
针对直接偏好优化忽略隐式奖励实际值的问题，提出Cal-DPO算法，通过校准隐式奖励规模使其与真实奖励相当，理论分析与多基准实验证明其显著提升了LLM与人类偏好的对齐效果。
摘要：
我们研究将大型语言模型（LLM）与人类偏好数据对齐的问题。对比偏好优化通过优化与策略相关的隐式奖励，在利用可用偏好数据对齐LLM方面展现出可喜的结果。然而，对比目标主要关注与两个响应相关的隐式奖励的相对值，而忽略了它们的实际值，导致与人类偏好的对齐效果次优。为了解决这一局限性，我们提出了校准直接偏好优化（Cal-DPO），这是一种简单而有效的算法。我们证明，只需对隐式奖励进行校准以确保学习到的隐式奖励在规模上与真实奖励相当，即可实现对给定偏好对齐的实质性改善。我们展示了Cal-DPO相对于现有方法的理论优势。我们在多种标准基准上的实验结果表明，Cal-DPO显著改进了现成方法。
英文摘要：
We study the problem of aligning large language models (LLMs) with human preference data. Contrastive preference optimization has shown promising results in aligning LLMs with available preference data by optimizing the implicit reward associated with the policy. However, the contrastive objective focuses mainly on the relative values of implicit rewards associated with two responses while ignoring their actual values, resulting in suboptimal alignment with human preferences. To address this limitation, we propose calibrated direct preference optimization (Cal-DPO), a simple yet effective algorithm. We show that substantial improvement in alignment with the given preferences can be achieved simply by calibrating the implicit reward to ensure that the learned implicit rewards are comparable in scale to the ground-truth rewards. We demonstrate the theoretical advantages of Cal-DPO over existing approaches. The results of our experiments on a variety of standard benchmarks show that Cal-DPO remarkably improves off-the-shelf methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
