Pre-DPO：通过引导参考模型提升直接偏好优化的数据利用
Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Model
AI总结：
Pre-DPO通过引入引导参考模型提升直接偏好优化的数据利用效率，有效提高大语言模型的性能表现。
摘要：
直接偏好优化（DPO）通过直接优化人类偏好简化了基于人类反馈的强化学习（RLHF）过程，用于大型语言模型（LLMs）。我们发现，在DPO训练过程中，参考模型起到数据权重调节器的作用。然而，DPO中常见的做法是将策略模型和参考模型初始化为相同，这会导致数据利用效率低下并限制性能上限。同时，简单偏好优化（SimPO）中缺乏参考模型会降低训练鲁棒性，并需要更严格的条件以防止灾难性遗忘。在本文中，我们提出了Pre-DPO，一种简单而有效的基于DPO的训练范式，通过利用引导参考模型来提升偏好优化性能。该参考模型能够预见到通过训练偏好数据可达到的最佳策略状态，作为引导机制，自适应地将更高权重分配给更适合模型的样本，将更低权重分配给不太适合的样本。在AlpacaEval 2.0和Arena-Hard v0.1基准测试中，广泛实验表明，Pre-DPO在不依赖外部模型或额外数据的情况下，一致提升了DPO和SimPO的性能。
英文摘要：
Direct Preference Optimization (DPO) simplifies reinforcement learning from human feedback (RLHF) for large language models (LLMs) by directly optimizing human preferences without an explicit reward model. We find that during DPO training, the reference model plays the role of a data weight adjuster. However, the common practice of initializing the policy and reference models identically in DPO can lead to inefficient data utilization and impose a performance ceiling. Meanwhile, the lack of a reference model in Simple Preference Optimization (SimPO) reduces training robustness and necessitates stricter conditions to prevent catastrophic forgetting. In this work, we propose Pre-DPO, a simple yet effective DPO-based training paradigm that enhances preference optimization performance by leveraging a guiding reference model. This reference model provides foresight into the optimal policy state achievable through the training preference data, serving as a guiding mechanism that adaptively assigns higher weights to samples more suitable for the model and lower weights to those less suitable. Extensive experiments on AlpacaEval 2.0 and Arena-Hard v0.1 benchmarks demonstrate that Pre-DPO consistently improves the performance of both DPO and SimPO, without relying on external models or additional data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
