arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2112.04857cs.LGstat.ML
收藏

压缩卷积神经网络模型冗余度的一种新度量

A New Measure of Model Redundancy for Compressed Convolutional Neural Networks

University of Hong Kong(香港大学);University of Connecticut(康涅狄格大学)

Feiqing Huang, Yuefeng Si, Yao Zheng, Guodong Li

摘要:

虽然最近已经提出了许多设计来提高卷积神经网络(CNN)在固定资源预算上的模型效率,但仍然明显缺乏对这些设计的理论理解。本文旨在提供一个新的框架来回答这个问题:在压缩的CNN中是否还有任何剩余的模型冗余?我们首先通过张量分解开发CNN和压缩CNN的一般统计公式,以便跨层的权重可以汇总到单个张量中。然后,通过严格的样本复杂性分析,我们揭示了一个重要的差异之间的派生样本的复杂性和天真的参数计数,这是一个直接的指标模型冗余。受这一发现的启发,我们为压缩CNN引入了一种新的模型冗余度量,称为$K/R$比,它进一步允许非线性激活。流行的块设计和数据集的消融研究支持这种新措施的有用性。

英文摘要:

While recently many designs have been proposed to improve the model efficiency of convolutional neural networks (CNNs) on a fixed resource budget, theoretical understanding of these designs is still conspicuously lacking. This paper aims to provide a new framework for answering the question: Is there still any remaining model redundancy in a compressed CNN? We begin by developing a general statistical formulation of CNNs and compressed CNNs via the tensor decomposition, such that the weights across layers can be summarized into a single tensor. Then, through a rigorous sample complexity analysis, we reveal an important discrepancy between the derived sample complexity and the naive parameter counting, which serves as a direct indicator of the model redundancy. Motivated by this finding, we introduce a new model redundancy measure for compressed CNNs, called the $K/R$ ratio, which further allows for nonlinear activations. The usefulness of this new measure is supported by ablation studies on popular block designs and datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑