BAPO：用于克服大语言模型个性化中遗忘问题的基座锚定偏好优化
BAPO: Base-Anchored Preference Optimization for Overcoming Forgetting in Large Language Models Personalization
AI总结：
本文提出BAPO方法，通过利用参考模型的初始响应作为锚点，在实现大语言模型个性化偏好对齐的同时有效缓解知识遗忘，实验验证了其在多种设置下的有效性。
摘要：
尽管让大语言模型（LLMs）与人类偏好对齐已取得显著成功，但使这些模型满足多样化的用户偏好在保留先前知识方面提出了进一步挑战。本文研究了面向个性化偏好的优化对LLMs的影响，揭示了知识损失程度随偏好异质性显著变化。虽然以往方法利用了参考模型与策略模型之间的KL约束，但我们观察到它们在面对个性化偏好时无法保持通用知识和对齐能力。为此，我们提出了基座锚定偏好优化（BAPO），这是一种简单而有效的方法，利用参考模型的初始响应来缓解遗忘，同时适应个性化对齐。BAPO能有效适应多样化的用户偏好，同时对全局知识或通用对齐的影响最小。我们的实验证明了BAPO在多种设置下的有效性。
英文摘要：
While learning to align Large Language Models (LLMs) with human preferences has shown remarkable success, aligning these models to meet the diverse user preferences presents further challenges in preserving previous knowledge. This paper examines the impact of personalized preference optimization on LLMs, revealing that the extent of knowledge loss varies significantly with preference heterogeneity. Although previous approaches have utilized the KL constraint between the reference model and the policy model, we observe that they fail to maintain general knowledge and alignment when facing personalized preferences. To this end, we introduce Base-Anchored Preference Optimization (BAPO), a simple yet effective approach that utilizes the initial responses of reference model to mitigate forgetting while accommodating personalized alignment. BAPO effectively adapts to diverse user preferences while minimally affecting global knowledge or general alignment. Our experiments demonstrate the efficacy of BAPO in various setups.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
