权重重采样与优化器如何塑造神经网络中持续学习与遗忘的动态过程
How Weight Resampling and Optimizers Shape the Dynamics of Continual Learning and Forgetting in Neural Networks
AI总结：
本文研究了权重重采样（zapping）和优化器选择对神经网络持续学习与遗忘动态的影响，发现二者均能显著改变模型迁移后的恢复速度及多任务间的协同与干扰模式。
摘要：
近期持续学习领域的研究强调了在神经网络最后一层重采样权重（“zapping”）所带来的有益效果。尽管实证结果表明该方法的有效性，但驱动这些改进的底层机制仍不清楚。在本工作中，我们详细研究了卷积神经网络在具有挑战性的设置（如持续学习和少样本迁移学习）下，使用手写字符和自然图像进行训练时，其内部发生的学习与遗忘模式。我们的实验表明，在训练过程中经历过 zapping 的模型能够更快地从迁移到新领域的冲击中恢复。此外，为了更好地观察多任务设置中持续学习的效果，我们测量了每个单独任务受到的影响。结果表明，不仅 zapping，优化器的选择也能深刻影响学习与遗忘的动态过程，导致模型在迁移时进行顺序学习的过程中出现复杂的任务间协同/干扰模式。
英文摘要：
Recent work in continual learning has highlighted the beneficial effect of resampling weights in the last layer of a neural network (``zapping"). Although empirical results demonstrate the effectiveness of this approach, the underlying mechanisms that drive these improvements remain unclear. In this work, we investigate in detail the pattern of learning and forgetting that take place inside a convolutional neural network when trained in challenging settings such as continual learning and few-shot transfer learning, with handwritten characters and natural images. Our experiments show that models that have undergone zapping during training more quickly recover from the shock of transferring to a new domain. Furthermore, to better observe the effect of continual learning in a multi-task setting we measure how each individual task is affected. This shows that, not only zapping, but the choice of optimizer can also deeply affect the dynamics of learning and forgetting, causing complex patterns of synergy/interference between tasks to emerge when the model learns sequentially at transfer time.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
