arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2011.10520cs.NE
收藏

重新考虑有效神经网络修剪的权重衰减

Rethinking Weight Decay For Efficient Neural Network Pruning

Hugo Tessier, Vincent Gripon, Mathieu Léonardon, Matthieu Arzel, Thomas Hannagan, David Bertrand

摘要:

修剪在20世纪80年代末被引入用于泛化目的,现在已经成为压缩深度神经网络的主要手段。尽管近几十年来有许多创新,修剪方法仍然面临着阻碍其性能或可扩展性的核心问题。从该领域的早期工作中汲取灵感,特别是使用权重衰减来实现稀疏性,我们引入了选择性权重衰减(SWD),它在整个训练过程中进行高效,连续的修剪。我们的方法,理论上基于拉格朗日平滑,是通用的,可以应用于多个任务,网络和修剪结构。我们表明,在CIFAR-10,Cora和ImageNet ILSVRC 2012数据集上,SWD在性能参数比方面优于最先进的方法。

英文摘要:

Introduced in the late 1980s for generalization purposes, pruning has now become a staple for compressing deep neural networks. Despite many innovations in recent decades, pruning approaches still face core issues that hinder their performance or scalability. Drawing inspiration from early work in the field, and especially the use of weight decay to achieve sparsity, we introduce Selective Weight Decay (SWD), which carries out efficient, continuous pruning throughout training. Our approach, theoretically grounded on Lagrangian smoothing, is versatile and can be applied to multiple tasks, networks, and pruning structures. We show that SWD compares favorably to state-of-the-art approaches, in terms of performance-to-parameters ratio, on the CIFAR-10, Cora, and ImageNet ILSVRC2012 datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑