GroupDPO：内存高效的组级直接偏好优化
GroupDPO: Memory-Efficient Group-Wise Direct Preference Optimization
AI总结：
本文提出GroupDPO，通过解耦样本和保留梯度实现高效的组级偏好优化，减少内存使用，提升大规模训练效果，且在离线和在线对齐中均优于单对训练。
摘要：
偏好优化广泛用于对齐大语言模型（LLMs）与偏好反馈。然而，现有方法通常针对每个提示训练一个正负对，丢弃了偏好数据集中通常包含多个候选响应的额外监督信息。受此限制启发，近期工作探索了组级偏好优化，通过联合对比相同提示的多个响应，但其经验行为和可扩展性因组耦合目标的内存开销而未被充分研究。本文提出一种内存高效的组级偏好优化算法，在反向传播过程中保留梯度并解耦样本，显著降低峰值内存使用，从而实现更大组大小的可扩展训练。在离线和在线对齐设置中，我们证明利用多个响应始终优于单对训练。此外，正响应上加入负对数似然（NLL）项对性能提升和训练稳定性至关重要。
英文摘要：
Preference optimization is widely used to align Large Language Models (LLMs) with preference feedback. However, most existing methods train on a single positive-negative pair per prompt, discarding additional supervision available in preference datasets that typically contain multiple candidate responses. Motivated by this limitation, recent work explores group-wise preference optimization, which jointly contrasts multiple responses for the same prompt, but its empirical behavior and scalability remain underexplored due to the memory overhead of group-coupled objectives. In this work, we present a unified empirical and systems study of group-wise preference optimization and develop a memory-efficient implementation for group-coupled objectives. By instantiating first-order linearization with objective-specific per-response coefficients, our implementation preserves first-order gradients while decoupling samples during backpropagation, substantially reducing peak memory usage and enabling scalable training with larger groups. Across offline and online settings, we show that leveraging multiple responses consistently outperforms single-pair training. Furthermore, incorporating a negative log-likelihood (NLL) term on positive responses is critical for both performance gains and training stability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
