LifeAlign: 为大型语言模型设计的终身对齐方法，结合记忆增强的聚焦偏好优化
LifeAlign: Lifelong Alignment for Large Language Models with Memory-Augmented Focalized Preference Optimization
AI总结：
本文提出LifeAlign，一种通过记忆增强的聚焦偏好优化实现终身对齐的方法，解决大型语言模型在连续学习任务中保持偏好对齐和知识保留的问题。
摘要：
对齐在大型语言模型（LLMs）中起关键作用，用于在特定任务/领域内对齐人类偏好。传统对齐方法存在灾难性遗忘问题，即当模型适应新偏好或领域时会失去之前获得的知识。我们引入LifeAlign，一种新的终身对齐框架，使LLMs能够在连续学习任务中保持一致的人类偏好对齐，而不会遗忘之前学习的知识。我们的方法包含两个关键创新。首先，我们提出了一种聚焦偏好优化策略，该策略在对齐新偏好时防止之前任务获得的知识的侵蚀。其次，我们开发了一种短到长记忆巩固机制，该机制将去噪的短期偏好表示合并到稳定的长期记忆中，利用内在维度约简，使在不同领域中高效存储和检索对齐模式成为可能。我们评估了LifeAlign在多个连续对齐任务上的表现，这些任务覆盖了不同的领域和偏好类型。实验结果表明，我们的方法在保持偏好对齐质量和知识保留方面优于现有的终身学习方法。代码和数据集已发布在https://github.com/real-ljs/LifeAlign。
英文摘要：
Alignment plays a crucial role in Large Language Models (LLMs) in aligning with human preferences on a specific task/domain. Traditional alignment methods suffer from catastrophic forgetting, where models lose previously acquired knowledge when adapting to new preferences or domains. We introduce LifeAlign, a novel framework for lifelong alignment that enables LLMs to maintain consistent human preference alignment across sequential learning tasks without forgetting previously learned knowledge. Our approach consists of two key innovations. First, we propose a focalized preference optimization strategy that aligns LLMs with new preferences while preventing the erosion of knowledge acquired from previous tasks. Second, we develop a short-to-long memory consolidation mechanism that merges denoised short-term preference representations into stable long-term memory using intrinsic dimensionality reduction, enabling efficient storage and retrieval of alignment patterns across diverse domains. We evaluate LifeAlign across multiple sequential alignment tasks spanning different domains and preference types. Experimental results demonstrate that our method achieves superior performance in maintaining both preference alignment quality and knowledge retention compared to existing lifelong learning approaches. The codes and datasets have been released on https://github.com/real-ljs/LifeAlign.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
