arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1811.09332cs.NE
收藏

预算感知正则化神经网络的结构化剪枝

Structured Pruning of Neural Networks with Budget-Aware Regularization

Carl Lemaire, Andrew Achkar, Pierre-Marc Jodoin

摘要:

修剪方法已被证明可以有效地减少深度神经网络的大小,同时保持准确性几乎不变。其中最有效的方法是修剪网络,同时使用稀疏先验损失和可学习的dropout参数进行训练。然而,这些方法的缺点是,修剪网络的大小和推理速度都不能直接控制;但这是在低功耗硬件上部署CNN的关键特征。为了克服这个问题,我们为深度CNN引入了一个预算正则化修剪框架。我们的方法自然适合传统的神经网络训练,因为它包括一个可学习的掩蔽层,一个新的知识感知目标函数,并使用知识蒸馏。我们还提供了关于如何修剪剩余网络以及如何产生新架构的见解。实验结果表明,使用我们的方法修剪的CNN比最先进的方法更准确,计算量更少。此外,我们的方法在严重修剪的情况下更有效地防止准确性崩溃;这使我们能够在没有显著准确性下降的情况下获得高达16倍的修剪因子。

英文摘要:

Pruning methods have shown to be effective at reducing the size of deep neural networks while keeping accuracy almost intact. Among the most effective methods are those that prune a network while training it with a sparsity prior loss and learnable dropout parameters. A shortcoming of these approaches however is that neither the size nor the inference speed of the pruned network can be controlled directly; yet this is a key feature for targeting deployment of CNNs on low-power hardware. To overcome this, we introduce a budgeted regularized pruning framework for deep CNNs. Our approach naturally fits into traditional neural network training as it consists of a learnable masking layer, a novel budget-aware objective function, and the use of knowledge distillation. We also provide insights on how to prune a residual network and how this can lead to new architectures. Experimental results reveal that CNNs pruned with our method are more accurate and less compute-hungry than state-of-the-art methods. Also, our approach is more effective at preventing accuracy collapse in case of severe pruning; this allows us to attain pruning factors up to 16x without significant accuracy drop.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑