基于直接偏好优化的LM对齐中，策略数据是否总是最佳选择？
Is On-Policy Data always the Best Choice for Direct Preference Optimization-based LM Alignment?
AI总结：
本文提出对齐阶段假设，通过理论和实证分析，揭示了静态与策略偏好数据在LM对齐中的效果差异，并提出算法识别对齐阶段边界。
摘要：
将语言模型（LMs）与人类偏好对齐对于构建可靠的人工智能系统至关重要。该问题通常被框架为优化LM策略以最大化反映人类偏好的期望奖励。最近，直接偏好优化（DPO）被提出作为LM对齐方法，该方法直接从静态偏好数据优化策略，并通过在训练循环中加入策略采样（即训练过程中生成的偏好候选）进一步改进，以实现更好的LM对齐。然而，我们证明策略数据并不总是最优的，静态和策略偏好候选之间会出现系统性的效果差异。例如，对于Llama-3，策略数据的效果是静态数据的3倍，而对于Zephyr则是0.4倍。为了解释这一现象，我们提出了对齐阶段假设，将对齐过程分为两个不同的阶段：偏好注入阶段，其受益于多样化数据；偏好微调阶段，其更青睐高质量数据。通过理论和实证分析，我们表征了这些阶段，并提出了一种有效的算法来识别它们之间的边界。我们在5个模型（Llama、Zephyr、Phi-2、Qwen、Pythia）和2种对齐方法（DPO、SLiC-HF）上进行了实验，以展示对齐阶段假设的一般性和边界测量算法的有效性。
英文摘要：
The alignment of language models~(LMs) with human preferences is critical for building reliable AI systems. The problem is typically framed as optimizing an LM policy to maximize the expected reward that reflects human preferences. Recently, Direct Preference Optimization~(DPO) was proposed as a LM alignment method that directly optimize the policy from static preference data, and further improved by incorporating on-policy sampling~(i.e., preference candidates generated during the training loop) for better LM alignment. However, we show on-policy data is not always optimal, with systematic effectiveness difference emerging between static and on-policy preference candidates. For example, on-policy data can result in a $3\times$ effectiveness compared with static data for Llama-3, and a $0.4\times$ effectiveness for Zephyr. To explain the phenomenon, we propose the alignment stage assumption, which divides the alignment process into two distinct stages: the preference injection stage, which benefits from diverse data, and the preference fine-tuning stage, which favors high-quality data. Through theoretical and empirical analysis, we characterize these stages and propose an effective algorithm to identify the boundaries between them. We perform experiments on $5$ models~(Llama, Zephyr, Phi-2, Qwen, Pythia) and $2$ alignment methods~(DPO, SLiC-HF) to show the generalizability of alignment stage assumption and the effectiveness of the boundary measurement algorithm.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
