arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2006.12463cs.CVcs.ITcs.LGmath.IT
收藏

使用自适应连通性分数精简神经网络

Slimming Neural Networks using Adaptive Connectivity Scores

University of Michigan(密歇根大学);University of Maine(缅因大学);Stevens Institute of Technology(史蒂文斯理工学院)

Madan Ravi Ganesh, Dawsin Blanchard, Jason J. Corso, Salimeh Yasaei Sekeh

摘要:

通常,深度神经网络(DNN)修剪方法分为两类:1)基于权重的确定性约束,以及2)概率框架。虽然每种方法都有其优点和局限性,但存在一系列常见的实际问题,例如分析灵敏度的试错和修剪DNN的超参数,这两种方法都困扰着它们。在这项工作中,我们提出了一种新的单次,全自动修剪算法称为瘦身神经网络使用自适应连接分数(SNACS)。我们提出的方法结合了一个概率修剪框架与约束的基础权重矩阵,通过一种新的连接性措施,在多个层次上利用这两种方法的优势,同时解决其不足之处。在\alg{}中,我们提出了一种快速的基于哈希的自适应条件互信息(ACMI)估计器,它使用一个基于权重的缩放标准,以评估过滤器之间的连接和修剪不重要的。为了自动确定层可以修剪的上限,我们提出了一组操作约束,这些约束联合定义了深度网络中所有层的修剪百分比上限。最后,我们定义了一个新的敏感性标准的过滤器,衡量其贡献的强度,以后续层和突出的关键过滤器,需要完全保护修剪。通过我们的实验验证,我们表明SNACS比最近的可比方法快17倍以上,并且是三个标准Dataset-DNN修剪基准的最先进的单次修剪方法:CIFAR 10-VGG 16,CIFAR 10-ResNet 56和ILSVRC 2012-ResNet 50。

英文摘要:

In general, deep neural network (DNN) pruning methods fall into two categories: 1) Weight-based deterministic constraints, and 2) Probabilistic frameworks. While each approach has its merits and limitations there are a set of common practical issues such as, trial-and-error to analyze sensitivity and hyper-parameters to prune DNNs, which plague them both. In this work, we propose a new single-shot, fully automated pruning algorithm called Slimming Neural networks using Adaptive Connectivity Scores (SNACS). Our proposed approach combines a probabilistic pruning framework with constraints on the underlying weight matrices, via a novel connectivity measure, at multiple levels to capitalize on the strengths of both approaches while solving their deficiencies. In \alg{}, we propose a fast hash-based estimator of Adaptive Conditional Mutual Information (ACMI), that uses a weight-based scaling criterion, to evaluate the connectivity between filters and prune unimportant ones. To automatically determine the limit up to which a layer can be pruned, we propose a set of operating constraints that jointly define the upper pruning percentage limits across all the layers in a deep network. Finally, we define a novel sensitivity criterion for filters that measures the strength of their contributions to the succeeding layer and highlights critical filters that need to be completely protected from pruning. Through our experimental validation we show that SNACS is faster by over 17x the nearest comparable method and is the state of the art single-shot pruning method across three standard Dataset-DNN pruning benchmarks: CIFAR10-VGG16, CIFAR10-ResNet56 and ILSVRC2012-ResNet50.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑