为何神经网络会遗忘：连续学习中崩溃的研究
Why Do Neural Networks Forget: A Study of Collapse in Continual Learning
AI总结：
本研究通过测量权重和激活的eRank，探讨了连续学习中遗忘与结构崩溃之间的关系，并评估了不同策略对模型容量和性能的影响。
摘要：
灾难性遗忘是连续学习中的主要问题，许多方法被提出以减少这种现象。然而，大多数方法都是通过任务准确性来评估的，这忽略了内部模型结构。最近的研究表明，结构崩溃导致了可塑性的丧失，这体现在有效秩（eRank）的变化上。这表明与遗忘有关，因为网络失去了扩展特征空间以学习新任务的能力，这迫使网络覆盖现有的表示。因此，在本研究中，我们通过测量权重和激活的eRank来调查遗忘与崩溃之间的相关性。具体而言，我们评估了四种架构，包括MLP、ConvGRU、ResNet-18和Bi-ConvGRU，在分割MNIST和分割CIFAR-100基准上。这些模型分别通过SGD、学习无遗忘（LwF）和经验回放（ER）策略进行训练。结果表明，遗忘与崩溃之间存在强烈关联，不同的连续学习策略在不同效率下帮助模型保持容量和性能。
英文摘要：
Catastrophic forgetting is a major problem in continual learning, and lots of approaches arise to reduce it. However, most of them are evaluated through task accuracy, which ignores the internal model structure. Recent research suggests that structural collapse leads to loss of plasticity, as evidenced by changes in effective rank (eRank). This indicates a link to forgetting, since the networks lose the ability to expand their feature space to learn new tasks, which forces the network to overwrite existing representations. Therefore, in this study, we investigate the correlation between forgetting and collapse through the measurement of both weight and activation eRank. To be more specific, we evaluated four architectures, including MLP, ConvGRU, ResNet-18, and Bi-ConvGRU, in the split MNIST and Split CIFAR-100 benchmarks. Those models are trained through the SGD, Learning-without-Forgetting (LwF), and Experience Replay (ER) strategies separately. The results demonstrate that forgetting and collapse are strongly related, and different continual learning strategies help models preserve both capacity and performance in different efficiency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
