重置并遗忘：重新学习最后一层权重可改善持续学习与迁移学习
Reset It and Forget It: Relearning Last-Layer Weights Improves Continual and Transfer Learning
AI总结：
本文提出反复重置最后一层权重的“zapping”预训练机制，在不依赖高阶梯度的情况下提升少样本迁移和持续学习性能，达到与先进元学习方法相当的效果。
摘要：
本工作识别出一种简单的预训练机制，该机制能够使表征展现出更好的持续学习和迁移学习能力。这一机制——反复重置最后一层权重，我们将其昵称为“zapping”——最初是为元持续学习过程设计的，但我们表明它在元学习和持续学习之外的许多场景中也出人意料地适用。在我们的实验中，我们希望将一个预训练的图像分类器迁移到一组新的类别上，且仅使用少量样本。我们表明，我们的 zapping 过程在标准微调和持续学习设置中都能带来更高的迁移准确率和/或更快速的适应，同时实现简单且计算高效。在许多情况下，通过结合使用 zapping 和顺序学习，我们能够在不依赖昂贵的高阶梯度的情况下，取得与最先进的元学习相当的性能。对这一 zapping 过程有效性的一个直观解释是：经过反复 zapping 训练的表征能够学习到可快速适应新初始化分类器的特征。这种方法可以被视为一种计算上更廉价的、或者替代使用高阶梯度来元学习快速适应特征的方法。这补充了近期关于在训练过程中重置神经网络参数的有用性的研究，并邀请对这一机制进行进一步探索。
英文摘要：
This work identifies a simple pre-training mechanism that leads to representations exhibiting better continual and transfer learning. This mechanism -- the repeated resetting of weights in the last layer, which we nickname "zapping" -- was originally designed for a meta-continual-learning procedure, yet we show it is surprisingly applicable in many settings beyond both meta-learning and continual learning. In our experiments, we wish to transfer a pre-trained image classifier to a new set of classes, in a few shots. We show that our zapping procedure results in improved transfer accuracy and/or more rapid adaptation in both standard fine-tuning and continual learning settings, while being simple to implement and computationally efficient. In many cases, we achieve performance on par with state of the art meta-learning without needing the expensive higher-order gradients, by using a combination of zapping and sequential learning. An intuitive explanation for the effectiveness of this zapping procedure is that representations trained with repeated zapping learn features that are capable of rapidly adapting to newly initialized classifiers. Such an approach may be considered a computationally cheaper type of, or alternative to, meta-learning rapidly adaptable features with higher-order gradients. This adds to recent work on the usefulness of resetting neural network parameters during training, and invites further investigation of this mechanism.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
