面向持续学习的记忆Transformer（Remembering Transformer）
Remembering Transformer for Continual Learning
AI总结：
针对持续学习中的灾难性遗忘问题，受大脑互补学习系统启发提出Remembering Transformer，通过适配器混合架构与新颖性检测机制结合知识蒸馏实现动态路由，在多类任务上取得SOTA性能且大幅降低内存占用。
摘要：
神经网络在持续学习中面临灾难性遗忘（Catastrophic Forgetting，CF）的挑战，即新任务的学习会干扰此前习得的知识。现有的数据微调和正则化方法在推理阶段需要任务身份信息，且无法消除不同任务间的干扰，而软参数共享方法则存在模型参数量持续增长的问题。为应对这些挑战，受大脑互补学习系统（Complementary Learning Systems，CLS）启发，我们提出了Remembering Transformer。该模型在预训练Transformer中采用适配器混合（mixture-of-adapters）架构以及基于生成模型的新颖性检测机制，以缓解灾难性遗忘。Remembering Transformer基于知识蒸馏，将任务数据动态路由至最相关的适配器，提升了参数效率。我们在广泛的类别增量拆分任务和排列任务上开展了大量实验，包括针对新颖性检测机制和适配器混合模型容量的消融研究。我们的方法取得了SOTA（state-of-the-art，当前最优）性能：在拆分任务中较次优方法提升15.90%，在五拆分CIFAR10任务中将内存占用从11.18M降至0.22M。
英文摘要：
Neural networks encounter the challenge of Catastrophic Forgetting (CF) in continual learning, where new task learning interferes with previously learned knowledge. Existing data fine-tuning and regularization methods necessitate task identity information during inference and cannot eliminate interference among different tasks, while soft parameter sharing approaches encounter the problem of an increasing model parameter size. To tackle these challenges, we propose the Remembering Transformer, inspired by the brain's Complementary Learning Systems (CLS). Remembering Transformer employs a mixture-of-adapters architecture and a generative model-based novelty detection mechanism in a pretrained Transformer to alleviate CF. Remembering Transformer dynamically routes task data to the most relevant adapter with enhanced parameter efficiency based on knowledge distillation. We conducted extensive experiments, including ablation studies on the novelty detection mechanism and model capacity of the mixture-of-adapters, in a broad range of class-incremental split tasks and permutation tasks. Our approach demonstrated SOTA performance surpassing the second-best method by 15.90% in the split tasks, reducing the memory footprint from 11.18M to 0.22M in the five splits CIFAR10 task.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
