arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2009.01395cs.LGstat.ML
收藏

网络压缩的部分正则化方法

A Partial Regularization Method for Network Compression

E Zhenqian, Gao Weiguo

摘要:

深度神经网络已经取得了显着的成功,这依赖于GPU和大规模数据集的发展,网络深度和宽度不断增加。然而,由于昂贵的计算和密集的内存,研究人员集中在设计压缩方法,以使它们适用于受限的平台。在本文中,我们提出了一种部分正则化的方法,而不是原来的形式,惩罚所有参数,这是所谓的完全正则化,进行模型压缩在一个更高的速度。根据神经网络的置换不变性的存在性,证明了该方法的合理性和可行性。实验结果表明,正如我们所预期的,计算复杂性降低观察到更少的运行时间在几乎所有的情况下。这是由于部分正则化方法所需计算单元数较少的缘故。令人惊讶的是,它有助于提高一些重要的指标,如回归拟合结果和多个数据集上的训练和测试阶段的分类精度,告诉我们修剪模型具有更好的性能和泛化能力。并对结果进行了分析,得出了一定存在最优网络结构的结论,最优网络结构依赖于输入数据。

英文摘要:

Deep Neural Networks have achieved remarkable success relying on the developing availability of GPUs and large-scale datasets with increasing network depth and width. However, due to the expensive computation and intensive memory, researchers have concentrated on designing compression methods in order to make them practical for constrained platforms. In this paper, we propose an approach of partial regularization rather than the original form of penalizing all parameters, which is said to be full regularization, to conduct model compression at a higher speed. It is reasonable and feasible according to the existence of the permutation invariant property of neural networks. Experimental results show that as we expected, the computational complexity is reduced by observing less running time in almost all situations. It should be owing to the fact that partial regularization method invovles a lower number of elements for calculation. Surprisingly, it helps to improve some important metrics such as regression fitting results and classification accuracy in both training and test phases on multiple datasets, telling us that the pruned models have better performance and generalization ability. What's more, we analyze the results and draw a conclusion that an optimal network structure must exist and depend on the input data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑