SAE-FD: 面向大语言模型持续学习的稀疏自编码器特征蒸馏
SAE-FD: Sparse Autoencoder Feature Distillation for Continual Learning of Large Language Models
AI总结：
针对持续学习中的灾难性遗忘问题，提出基于稀疏自编码器特征蒸馏的方法，通过将模型表示锚定在稀疏特征空间以减少表征纠缠，实现更精准的正则化，在多个基准上优于现有方法。
摘要：
持续学习使大语言模型能够适应不断变化的任务而无需从头重新训练，但灾难性遗忘仍然是一个核心障碍。在持续学习方法中，基于正则化的方法被广泛用于约束模型更新并减少遗忘，这些方法在权重空间、梯度空间或输出空间中操作。然而，这些密集表示空间存在特征叠加问题，即多个概念被编码在重叠的维度中，使得难以在不阻碍新任务学习的情况下有选择地保护先前学到的知识。为了解决这个问题，我们提出了\method（稀疏自编码器特征蒸馏），该方法将模型表示锚定在预训练稀疏自编码器的稀疏特征空间中，其中密集激活被分解为稀疏过完备基，从而减少表征纠缠，实现更有针对性的正则化，同时减少对新任务学习的干扰。在三个模型架构上的两个持续学习基准实验表明，\method始终优于现有的基于正则化的方法，平均准确率高达52.70%，仅产生-0.46的后向迁移。
英文摘要：
Continual learning enables large language models to adapt to evolving tasks without retraining from scratch, yet catastrophic forgetting remains a central obstacle. Among continual learning methods, regularization-based approaches are widely used to constrain model updates and reduce forgetting, operating in weight space, gradient space, or output space. However, these dense representation spaces suffer from feature superposition, where multiple concepts are encoded in overlapping dimensions, making it difficult to selectively protect previously learned knowledge without impeding new-task learning. To address this issue, we propose \method (Sparse Autoencoder Feature Distillation), which anchors model representations in the sparse feature space of a pre-trained Sparse Autoencoder, where dense activations are decomposed into a sparse overcomplete basis that reduces representational entanglement, enabling more targeted regularization with less interference to new-task learning. Experiments on two continual learning benchmarks across three model architectures show that \method consistently outperforms existing regularization-based methods, achieving up to 52.70% average accuracy with only -0.46 backward transfer.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
