基于注意力的学会学习且不遗忘方法
Learning to Learn without Forgetting using Attention
AI总结：
针对机器学习持续学习中易遗忘旧知识的问题，提出用元学习训练基于Transformer、带注意力机制的优化器，在基准数据集上验证了其在前后向迁移上的有效性。
摘要：
持续学习（Continual Learning，CL）指的是随着时间推移不断学习，在容纳新知识的同时保留先前习得经验的能力。尽管这一概念是人类学习的固有特性，但当前的机器学习方法极易覆盖先前习得的模式，从而遗忘过往经验。与之相对，模型参数应当被有选择性地、谨慎地更新，在避免不必要遗忘的同时，最优地利用先前习得的模式来加速未来的学习。由于手工设计有效的更新机制十分困难，我们提出通过元学习一个基于Transformer的优化器来增强持续学习效果。这个元学习得到的优化器利用注意力机制学习任务流中模型参数之间的复杂关系，其设计目标是为当前任务生成有效的权重更新，同时防止在先前遇到的任务上发生灾难性遗忘。在SplitMNIST、RotatedMNIST和SplitCIFAR-100等基准数据集上的评估证实，该方法在前向迁移和反向迁移两方面均具备有效性，即便在标记数据量很少的情况下也是如此，凸显了将元学习优化器整合进持续学习框架的优势。
英文摘要：
Continual learning (CL) refers to the ability to continually learn over time by accommodating new knowledge while retaining previously learned experience. While this concept is inherent in human learning, current machine learning methods are highly prone to overwrite previously learned patterns and thus forget past experience. Instead, model parameters should be updated selectively and carefully, avoiding unnecessary forgetting while optimally leveraging previously learned patterns to accelerate future learning. Since hand-crafting effective update mechanisms is difficult, we propose meta-learning a transformer-based optimizer to enhance CL. This meta-learned optimizer uses attention to learn the complex relationships between model parameters across a stream of tasks, and is designed to generate effective weight updates for the current task while preventing catastrophic forgetting on previously encountered tasks. Evaluations on benchmark datasets like SplitMNIST, RotatedMNIST, and SplitCIFAR-100 affirm the efficacy of the proposed approach in terms of both forward and backward transfer, even on small sets of labeled data, highlighting the advantages of integrating a meta-learned optimizer within the continual learning framework.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
