重新初始化权重与单元以维持神经网络的可塑性
Reinitializing weights vs units for maintaining plasticity in neural networks
AI总结：
针对神经网络在非平稳数据长时间训练后出现的可塑性丧失问题，本文提出选择性权重重新初始化算法，对比单元与权重重新初始化方案，发现前者在单元少或含层归一化场景更优，适用场景更广泛。
摘要：
可塑性丧失是指神经网络在非平稳数据上长时间训练后失去学习能力的现象。在设计持续学习系统时，这是一个需要克服的关键问题。防止可塑性丧失的有效技术是重新初始化网络的部分结构。本文比较了两种不同的重新初始化方案：重新初始化单元与重新初始化权重。我们提出了一种新算法，命名为\textit{选择性权重重新初始化}，用于重新初始化网络中最无用的权重。我们将该算法与持续反向传播（continual backpropagation）和ReDo这两种先前提出的重新初始化网络单元的算法进行比较。通过在持续监督学习问题中的实验，我们确定了两种重新初始化权重比重新初始化单元更能有效维持可塑性的场景：（1）当网络单元数量较少时；（2）当网络包含层归一化时。相反，当网络规模足够大且不包含层归一化时，重新初始化权重和单元在维持可塑性方面效果相当。我们发现，与重新初始化单元相比，重新初始化权重在更多样化的场景中能维持可塑性。
英文摘要：
Loss of plasticity is a phenomenon in which a neural network loses its ability to learn when trained for an extended time on non-stationary data. It is a crucial problem to overcome when designing systems that learn continually. An effective technique for preventing loss of plasticity is reinitializing parts of the network. In this paper, we compare two different reinitialization schemes: reinitializing units vs reinitializing weights. We propose a new algorithm, which we name \textit{selective weight reinitialization}, for reinitializing the least useful weights in a network. We compare our algorithm to continual backpropagation and ReDo, two previously proposed algorithms that reinitialize units in the network. Through our experiments in continual supervised learning problems, we identify two settings when reinitializing weights is more effective at maintaining plasticity than reinitializing units: (1) when the network has a small number of units and (2) when the network includes layer normalization. Conversely, reinitializing weights and units are equally effective at maintaining plasticity when the network is of sufficient size and does not include layer normalization. We found that reinitializing weights maintains plasticity in a wider variety of settings than reinitializing units.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
