通过前缀共享加速直接偏好优化
Accelerating Direct Preference Optimization with Prefix Sharing
AI总结：
针对离线成对偏好优化传统实现的冗余计算问题，提出前缀共享技术，配合块稀疏注意力掩码，在DPO数据集上实现1.1-1.5倍训练吞吐量提升，结合序列打包可达1.3-1.6倍加速，且适用于其他成对偏好调优方法。
摘要：
离线成对偏好优化算法已成为偏好数据微调的主流方法，在各类任务中表现优于传统监督微调。然而传统实现常存在冗余计算，尤其在共享提示词较长的任务中问题突出。我们提出面向偏好调优的前缀共享技术，将选中响应和拒绝响应作为带有共享前缀的单条序列处理，为避免跨响应污染，采用定制的块稀疏注意力掩码。该方法在主流DPO数据集上实现了1.1-1.5倍的训练吞吐量提升，且对收敛性无任何影响。与序列打包技术结合后，可稳定获得1.3-1.6倍的加速，甚至对序列长度较短的数据集也有效。尽管本研究聚焦于直接偏好优化（DPO），该方法同样适用于其他成对偏好调优方法。通过提升计算效率，本研究推动基于偏好的微调技术适配更广泛的应用场景与模型规模。我们已在https://github.com/frankxwang/dpo-prefix-sharing开源代码。
英文摘要：
Offline paired preference optimization algorithms have become a popular approach for fine-tuning on preference data, outperforming traditional supervised fine-tuning in various tasks. However, traditional implementations often involve redundant computations, especially for tasks with long shared prompts. We introduce prefix sharing for preference tuning, a novel technique that processes chosen and rejected responses as one sequence with a shared prefix. To prevent cross-response contamination, we use a custom block-sparse attention mask. Our method achieves $1.1$-$1.5\times$ improvement in training throughput on popular DPO datasets, without any effect on convergence. When combined with sequence packing, we observe consistent $1.3$-$1.6\times$ speedups, benefiting even datasets with smaller sequence lengths. While we focus on Direct Preference Optimization (DPO), our approach is applicable to other paired preference tuning methods. By enhancing computational efficiency, our work contributes to making preference-based fine-tuning more accessible for a wider range of applications and model sizes. We open-source our code at https://github.com/frankxwang/dpo-prefix-sharing.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
