Powerpropagation:一种稀疏诱导权重重参数化方法
Powerpropagation: A sparsity inducing weight reparameterisation
摘要:
稀疏神经网络的训练正在成为一种越来越重要的工具,用于减少模型在训练和评估时的计算足迹,以及实现模型的有效扩展。虽然多年来的大量工作一直致力于专门的修剪技术,很少注意到基于梯度的训练对模型稀疏性的固有影响。在这项工作中,我们介绍Powerpropagation,一个新的权重参数化的神经网络,导致固有的稀疏模型。利用梯度下降的行为,我们的方法产生了权重更新,表现出“富人更富”的动态,使低幅度的参数基本上不受学习的影响。以这种方式训练的模型表现出类似的性能,但具有在零处具有显著更高密度的分布,允许安全地修剪更多参数。Powerpropagation是通用的,直观的,便宜的和简单的实现,可以很容易地与各种其他技术相结合。为了突出它的通用性,我们在两个非常不同的环境中探索它:首先,根据最近的工作,我们研究了它对资源受限环境下稀疏训练的影响。在这里,我们将Powerpropagation与传统的权重修剪技术以及最新的最先进的稀疏到稀疏算法相结合,在ImageNet基准测试中表现出卓越的性能。其次,我们提倡使用稀疏性来克服灾难性遗忘,压缩表示允许以固定的模型容量容纳大量的任务。在所有情况下,我们的重新参数化大大提高了现成方法的有效性。
英文摘要:
The training of sparse neural networks is becoming an increasingly important tool for reducing the computational footprint of models at training and evaluation, as well enabling the effective scaling up of models. Whereas much work over the years has been dedicated to specialised pruning techniques, little attention has been paid to the inherent effect of gradient based training on model sparsity. In this work, we introduce Powerpropagation, a new weight-parameterisation for neural networks that leads to inherently sparse models. Exploiting the behaviour of gradient descent, our method gives rise to weight updates exhibiting a "rich get richer" dynamic, leaving low-magnitude parameters largely unaffected by learning. Models trained in this manner exhibit similar performance, but have a distribution with markedly higher density at zero, allowing more parameters to be pruned safely. Powerpropagation is general, intuitive, cheap and straight-forward to implement and can readily be combined with various other techniques. To highlight its versatility, we explore it in two very different settings: Firstly, following a recent line of work, we investigate its effect on sparse training for resource-constrained settings. Here, we combine Powerpropagation with a traditional weight-pruning technique as well as recent state-of-the-art sparse-to-sparse algorithms, showing superior performance on the ImageNet benchmark. Secondly, we advocate the use of sparsity in overcoming catastrophic forgetting, where compressed representations allow accommodating a large number of tasks at fixed model capacity. In all cases our reparameterisation considerably increases the efficacy of the off-the-shelf methods.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。