基于循环投影的深度网络参数冗余研究
An exploration of parameter redundancy in deep networks with circulant projections
摘要：
我们通过用循环投影代替传统的全连接层线性投影来探索深度神经网络中参数的冗余性。循环结构大大减少了内存占用，并使快速傅立叶变换的使用，以加快计算。该方法考虑一个全连接的神经网络层，其输入节点为d，输出节点为d，时间复杂度从O（d^2）降为O（dlogd），空间复杂度空间复杂度从O（d^2）降为O（d）。空间节省对于现代深度卷积神经网络架构尤其重要，其中完全连接的层通常包含超过90%的网络参数。我们进一步表明，梯度计算和优化的循环投影可以非常有效地进行。我们在三个标准数据集上的实验表明，与具有非结构化投影的神经网络相比，所提出的方法在存储和效率方面实现了显着的增益，并且出错率增加最小。
英文摘要：
We explore the redundancy of parameters in deep neural networks by replacing the conventional linear projection in fully-connected layers with the circulant projection. The circulant structure substantially reduces memory footprint and enables the use of the Fast Fourier Transform to speed up the computation. Considering a fully-connected neural network layer with d input nodes, and d output nodes, this method improves the time complexity from O(d^2) to O(dlogd) and space complexity from O(d^2) to O(d). The space savings are particularly important for modern deep convolutional neural network architectures, where fully-connected layers typically contain more than 90% of the network parameters. We further show that the gradient computation and optimization of the circulant projections can be performed very efficiently. Our experiments on three standard datasets show that the proposed approach achieves this significant gain in storage and efficiency with minimal increase in error rate compared to neural networks with unstructured projections.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
