arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2108.00065cs.LG
收藏

神经网络的保模型压缩

Model Preserving Compression for Neural Networks

Cornell University(康奈尔大学)

Jerry Chee, Megan Renz, Anil Damle, Christopher De Sa

摘要:

在训练复杂的深度学习模型后,一个常见的任务是压缩模型以减少计算和存储需求。当压缩时,期望保留原始模型的每个示例的决策(例如,超越top-1精度或保持鲁棒性)、维持网络结构、自动确定每层压缩级别,并消除对微调的需要。没有现有的压缩方法同时满足这些标准$\unicode{x2014}$,我们引入了一个原则性的方法,利用插值分解。我们的方法同时选择和消除通道(类似于神经元),然后构建一个插值矩阵,将校正传播到下一层,保留网络的结构。因此,我们的方法实现了良好的性能,即使没有微调,并承认理论分析。我们对单层网络的理论概括自然适用于启发式方法,该方法允许我们的方法自动选择深层网络的每层大小。我们证明了我们的方法的有效性,在各种任务,模型和数据集$\unicode{x2014}$上具有强大的经验性能,从简单的一个隐藏层网络到ImageNet上的深度网络。

英文摘要:

After training complex deep learning models, a common task is to compress the model to reduce compute and storage demands. When compressing, it is desirable to preserve the original model's per-example decisions (e.g., to go beyond top-1 accuracy or preserve robustness), maintain the network's structure, automatically determine per-layer compression levels, and eliminate the need for fine tuning. No existing compression methods simultaneously satisfy these criteria $\unicode{x2014}$ we introduce a principled approach that does by leveraging interpolative decompositions. Our approach simultaneously selects and eliminates channels (analogously, neurons), then constructs an interpolation matrix that propagates a correction into the next layer, preserving the network's structure. Consequently, our method achieves good performance even without fine tuning and admits theoretical analysis. Our theoretical generalization bound for a one layer network lends itself naturally to a heuristic that allows our method to automatically choose per-layer sizes for deep networks. We demonstrate the efficacy of our approach with strong empirical performance on a variety of tasks, models, and datasets $\unicode{x2014}$ from simple one-hidden-layer networks to deep networks on ImageNet.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑