在线DPO：具有快慢追逐机制的在线直接偏好优化
Online DPO: Online Direct Preference Optimization with Fast-Slow Chasing
AI总结：
本文提出在线快慢追逐DPO（OFS-DPO）及其跨域扩展COFS-DPO，通过双LoRA模块模拟种内竞争并利用历史参数组合，缓解跨域持续偏好对齐中的灾难性遗忘问题。
摘要：
直接偏好优化（DPO）通过直接在人类偏好数据集上训练来改善大语言模型（LLMs）与人类价值观的对齐，从而无需奖励模型。然而，由于跨域人类偏好的存在，直接持续训练可能导致灾难性遗忘，限制了DPO的性能和效率。受种内竞争驱动物种进化的启发，我们提出了一种用于偏好对齐的在线快慢追逐DPO（OFS-DPO），通过模型之间的快慢追逐模拟竞争以促进快速适应。具体来说，我们首先推导出在线学习的遗憾上界，用最小-最大优化模式验证了我们的动机。在此基础上，我们引入两个使用低秩自适应（LoRA）且优化速度不同的相同模块来模拟种内竞争，并提出一个新的正则化项来引导它们的学习。为了进一步缓解跨域场景中的灾难性遗忘，我们将OFS-DPO扩展为LoRA模块组合策略，得到跨域在线快慢追逐DPO（COFS-DPO）。该方法利用来自不同任务域的快速模块参数的线性组合，充分利用历史信息实现持续价值对齐。实验结果表明，OFS-DPO在域内对齐方面优于DPO，而COFS-DPO在跨域持续学习场景中表现优异。
英文摘要：
Direct Preference Optimization (DPO) improves the alignment of large language models (LLMs) with human values by training directly on human preference datasets, eliminating the need for reward models. However, due to the presence of cross-domain human preferences, direct continual training can lead to catastrophic forgetting, limiting DPO's performance and efficiency. Inspired by intraspecific competition driving species evolution, we propose a Online Fast-Slow chasing DPO (OFS-DPO) for preference alignment, simulating competition through fast and slow chasing among models to facilitate rapid adaptation. Specifically, we first derive the regret upper bound for online learning, validating our motivation with a min-max optimization pattern. Based on this, we introduce two identical modules using Low-rank Adaptive (LoRA) with different optimization speeds to simulate intraspecific competition, and propose a new regularization term to guide their learning. To further mitigate catastrophic forgetting in cross-domain scenarios, we extend the OFS-DPO with LoRA modules combination strategy, resulting in the Cross domain Online Fast-Slow chasing DPO (COFS-DPO). This method leverages linear combinations of fast modules parameters from different task domains, fully utilizing historical information to achive continual value alignment. Experimental results show that OFS-DPO outperforms DPO in in-domain alignment, while COFS-DPO excels in cross-domain continual learning scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
