关于神经网络中剩余容量的度量
On Measuring Excess Capacity in Neural Networks
摘要：
我们在监督分类的背景下研究了深度网络的过剩容量。也就是说，给定一个潜在假设类的能力度量--在我们的例子中，是经验的拉德马赫复杂性--在多大程度上，我们可以（先验地）约束这个类，同时保留与无约束机制相当的经验误差？为了评估现代架构（如剩余网络）中的过剩容量，我们扩展并统一了先前的Rademacher复杂性界限，以适应函数的合成和加法，以及卷积的结构。我们边界中的容量驱动项是层的Lipschitz常数和到卷积权重初始化的（2，1）群范数距离。在不同任务难度的基准数据集上进行的实验表明：（1）每个任务都有大量的过剩容量，（2）跨任务的容量可以保持在惊人的相似水平。总的来说，这表明了关于权重范数的可压缩性的概念，与通过权重修剪的经典压缩互补。源代码可在https://github.com/rkwitt/excess_capacity上获得。
英文摘要：
We study the excess capacity of deep networks in the context of supervised classification. That is, given a capacity measure of the underlying hypothesis class - in our case, empirical Rademacher complexity - to what extent can we (a priori) constrain this class while retaining an empirical error on a par with the unconstrained regime? To assess excess capacity in modern architectures (such as residual networks), we extend and unify prior Rademacher complexity bounds to accommodate function composition and addition, as well as the structure of convolutions. The capacity-driving terms in our bounds are the Lipschitz constants of the layers and an (2, 1) group norm distance to the initializations of the convolution weights. Experiments on benchmark datasets of varying task difficulty indicate that (1) there is a substantial amount of excess capacity per task, and (2) capacity can be kept at a surprisingly similar level across tasks. Overall, this suggests a notion of compressibility with respect to weight norms, complementary to classic compression via weight pruning. Source code is available at https://github.com/rkwitt/excess_capacity.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
