Eidetic学习：一种高效且可证明的灾难性遗忘解决方案
Eidetic Learning: an Efficient and Provable Solution to Catastrophic Forgetting
AI总结：
针对神经网络灾难性遗忘问题，提出无需复述或回放的Eidetic Learning方法，所训练的EideticNet可自动路由新实例，保证免疫遗忘，且时空复杂度线性，适用于现代网络归一化层，实践与理论价值兼具。
摘要：
灾难性遗忘——神经网络学习任务t1后，在训练其他任务t2时失去执行t1能力的现象——是神经网络长期存在的问题[McCloskey and Cohen, 1989]。我们提出Eidetic学习方法，可证明解决该问题。经Eidetic学习训练的网络（EideticNet）无需复述或回放。我们考虑连续离散任务，表明推理时EideticNet无需辅助任务信息即可自动路由新实例。EideticNet与稀疏门控混合专家层[Shazeer et al., 2016]有家族相似性，即网络容量按任务划分，且网络自身执行数据条件路由。EideticNet易于实现和训练，高效，时空复杂度与参数数量呈线性关系。我们的方法保证适用于现代神经网络的归一化层，无论预训练还是微调阶段。通过多种网络架构和任务集，我们证明EideticNet免疫遗忘。尽管其实际益处显著，我们相信它能同时造福实践者和理论家。训练代码可在https://github.com/amazon-science/eideticnet-training获取。
英文摘要：
Catastrophic forgetting -- the phenomenon of a neural network learning a task t1 and losing the ability to perform it after being trained on some other task t2 -- is a long-standing problem for neural networks [McCloskey and Cohen, 1989]. We present a method, Eidetic Learning, that provably solves catastrophic forgetting. A network trained with Eidetic Learning -- here, an EideticNet -- requires no rehearsal or replay. We consider successive discrete tasks and show how at inference time an EideticNet automatically routes new instances without auxiliary task information. An EideticNet bears a family resemblance to the sparsely-gated Mixture-of-Experts layer Shazeer et al. [2016] in that network capacity is partitioned across tasks and the network itself performs data-conditional routing. An EideticNet is easy to implement and train, is efficient, and has time and space complexity linear in the number of parameters. The guarantee of our method holds for normalization layers of modern neural networks during both pre-training and fine-tuning. We show with a variety of network architectures and sets of tasks that EideticNets are immune to forgetting. While the practical benefits of EideticNets are substantial, we believe they can be benefit practitioners and theorists alike. The code for training EideticNets is available at https://github.com/amazon-science/eideticnet-training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
