arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1507.05775cs.NEcs.CVcs.LG
收藏

用克罗内克积压缩神经网络全连接层

Compression of Fully-Connected Layer in Neural Network by Kronecker Product

Institute of Computing Technology(计算技术研究所);Chinese Academy of Sciences(中国科学院);Megvii Inc.(旷视科技有限公司);Shanghai Jiao Tong University(上海交通大学)

Shuchang Zhou, Jia-Nan Wu

摘要:

在本文中,我们提出并研究了一种技术,以减少参数的数量和计算时间在全连接层的神经网络使用克罗内克积,在一个温和的成本的预测质量。该技术通过用所谓的克罗内克全连接层替换全连接层来进行,其中FC层的权重矩阵由较小矩阵的多个克罗内克乘积的线性组合来近似。特别是,给定一个在SVHN数据集上训练的模型,我们能够构建一个新的KFC模型,参数总数减少了73%,而误差只轻微上升。相比之下,使用低秩方法只能实现35%的总数量的参数减少相同的质量退化容限。如果我们仅比较KFC层与其对应的全连接层,则参数数量的减少超过99%。由于我们用KFC层中几个较小矩阵的矩阵乘积代替FC层中大矩阵的矩阵乘积,计算量也减少了。在MNIST、SVHN和一些汉字识别模型上的实验也证明了该方法的有效性。

英文摘要:

In this paper we propose and study a technique to reduce the number of parameters and computation time in fully-connected layers of neural networks using Kronecker product, at a mild cost of the prediction quality. The technique proceeds by replacing Fully-Connected layers with so-called Kronecker Fully-Connected layers, where the weight matrices of the FC layers are approximated by linear combinations of multiple Kronecker products of smaller matrices. In particular, given a model trained on SVHN dataset, we are able to construct a new KFC model with 73\% reduction in total number of parameters, while the error only rises mildly. In contrast, using low-rank method can only achieve 35\% reduction in total number of parameters given similar quality degradation allowance. If we only compare the KFC layer with its counterpart fully-connected layer, the reduction in the number of parameters exceeds 99\%. The amount of computation is also reduced as we replace matrix product of the large matrices in FC layers with matrix products of a few smaller matrices in KFC layers. Further experiments on MNIST, SVHN and some Chinese Character recognition models also demonstrate effectiveness of our technique.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑