用于持续学习的部分超网络
Partial Hypernetworks for Continual Learning
AI总结：
本文提出部分超网络方法，仅生成模型最后几层权重并冻结初始层，以缓解持续学习中的遗忘问题，在CIFAR-100和TinyImagenet上验证了其计算与准确率的有效平衡。
摘要：
超网络通过生成任务相关的权重并在元模型层面惩罚权重变化，从而缓解持续学习（CL）中的遗忘问题。然而，对于较大的架构，生成所有权重不仅在计算上昂贵，而且生成所有模型权重是否必要也尚不清楚。受持续学习中潜在回放方法的启发，我们提出使用超网络仅生成模型最后几层的部分权重，同时冻结初始层。基于这一目标，我们首先回答了在不影响最终性能的情况下可以冻结多少层的问题。通过多项实验，我们经验性地表明，可冻结的层数与持续学习流中的分布相似性成正比。然后，为证明超网络的有效性，我们展示了噪声流会显著影响潜在回放方法的性能，导致当来自噪声经验的特征与旧样本一起回放时遗忘增加。相比之下，部分超网络通过保持先前经验的准确性而对噪声更具鲁棒性。最后，我们在分割的CIFAR-100和TinyImagenet基准上进行了实验，并将不同版本的部分超网络与潜在回放方法进行了比较。我们得出结论，使用超网络进行部分权重生成是解决神经网络遗忘问题的一个有前景的方案。它可以在持续学习流中提供计算与最终测试准确率之间的有效平衡。
英文摘要：
Hypernetworks mitigate forgetting in continual learning (CL) by generating task-dependent weights and penalizing weight changes at a meta-model level. Unfortunately, generating all weights is not only computationally expensive for larger architectures, but also, it is not well understood whether generating all model weights is necessary. Inspired by latent replay methods in CL, we propose partial weight generation for the final layers of a model using hypernetworks while freezing the initial layers. With this objective, we first answer the question of how many layers can be frozen without compromising the final performance. Through several experiments, we empirically show that the number of layers that can be frozen is proportional to the distributional similarity in the CL stream. Then, to demonstrate the effectiveness of hypernetworks, we show that noisy streams can significantly impact the performance of latent replay methods, leading to increased forgetting when features from noisy experiences are replayed with old samples. In contrast, partial hypernetworks are more robust to noise by maintaining accuracy on previous experiences. Finally, we conduct experiments on the split CIFAR-100 and TinyImagenet benchmarks and compare different versions of partial hypernetworks to latent replay methods. We conclude that partial weight generation using hypernetworks is a promising solution to the problem of forgetting in neural networks. It can provide an effective balance between computation and final test accuracy in CL streams.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
