在深度持续学习中保持可塑性
Maintaining Plasticity in Deep Continual Learning
AI总结：
本文研究深度持续学习中的可塑性丧失问题，通过 MNIST 与 ImageNet 任务序列展示其普遍存在，并提出 continual backpropagation 以持续重初始化低使用率单元来长期保持可塑性。
摘要：
现代深度学习系统专门适用于训练只发生一次、此后不再发生的问题设定，而不适用于训练持续进行的持续学习设定。如果将深度学习系统应用于持续学习设定，众所周知，它们可能无法记住较早的样本。更根本但较少为人知的是，它们还可能丧失学习新样本的能力，这一现象被称为可塑性丧失。我们使用被改造为任务序列以用于持续学习的 MNIST 和 ImageNet 数据集，对可塑性丧失进行了直接演示。在 ImageNet 中，二分类性能从早期任务上 89% 的准确率下降到第 2000 个任务上的 77%，约为线性网络的水平。可塑性丧失在多种深度网络架构、优化器、激活函数、batch normalization 和 dropout 下都会发生，但通过 L2 正则化可显著缓解，尤其是在与权重扰动结合时。此外，我们提出了一种新算法——continual backpropagation（持续反向传播），它对传统反向传播稍作修改，在每个样本之后重新初始化一小部分较少使用的单元，并且似乎能够无限期地保持可塑性。
英文摘要：
Modern deep-learning systems are specialized to problem settings in which training occurs once and then never again, as opposed to continual-learning settings in which training occurs continually. If deep-learning systems are applied in a continual learning setting, then it is well known that they may fail to remember earlier examples. More fundamental, but less well known, is that they may also lose their ability to learn on new examples, a phenomenon called loss of plasticity. We provide direct demonstrations of loss of plasticity using the MNIST and ImageNet datasets repurposed for continual learning as sequences of tasks. In ImageNet, binary classification performance dropped from 89% accuracy on an early task down to 77%, about the level of a linear network, on the 2000th task. Loss of plasticity occurred with a wide range of deep network architectures, optimizers, activation functions, batch normalization, dropout, but was substantially eased by L2-regularization, particularly when combined with weight perturbation. Further, we introduce a new algorithm -- continual backpropagation -- which slightly modifies conventional backpropagation to reinitialize a small fraction of less-used units after each example and appears to maintain plasticity indefinitely.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
