arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2107.05787cs.LG
收藏

数据驱动的低秩神经网络压缩

Data-Driven Low-Rank Neural Network Compression

UC Berkeley(加州大学伯克利分校);InterDigital AI Lab(InterDigital人工智能实验室);Technicolor AI Lab(特艺人工智能实验室)

Dimitris Papadimitriou, Swayambhoo Jain

摘要:

尽管深度神经网络(DNN)有许多现代应用,但隐藏层中的大量参数使得它们对于在具有存储容量限制的设备上部署没有吸引力。在本文中,我们提出了一种数据驱动的低秩(DDLR)方法,通过在全连接层上施加低秩结构来减少预训练DNN的参数数量并加快推理,同时控制整体准确性并且不需要任何重新训练。我们提出的问题,找到最低的秩近似的每个全连接层与给定的性能保证,并放松到一个易于处理的凸优化问题。我们表明,可以显着减少常见DNN架构中的参数数量,而分类精度只会有很小的降低。我们将DDLR与Net-Trim进行了比较,Net-Trim是另一种基于稀疏性的数据驱动DNN压缩技术,并表明DDLR在保持更高准确性的同时始终产生更多压缩的神经网络。

英文摘要:

Despite many modern applications of Deep Neural Networks (DNNs), the large number of parameters in the hidden layers makes them unattractive for deployment on devices with storage capacity constraints. In this paper we propose a Data-Driven Low-rank (DDLR) method to reduce the number of parameters of pretrained DNNs and expedite inference by imposing low-rank structure on the fully connected layers, while controlling for the overall accuracy and without requiring any retraining. We pose the problem as finding the lowest rank approximation of each fully connected layer with given performance guarantees and relax it to a tractable convex optimization problem. We show that it is possible to significantly reduce the number of parameters in common DNN architectures with only a small reduction in classification accuracy. We compare DDLR with Net-Trim, which is another data-driven DNN compression technique based on sparsity and show that DDLR consistently produces more compressed neural networks while maintaining higher accuracy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑