arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1905.12430cs.LGstat.ML
收藏

多类卷积神经网络基于范数的泛化界

Norm-based generalisation bounds for multi-class convolutional neural networks

TU Kaiserslautern(凯泽斯劳滕工业大学);University of Birmingham(伯明翰大学)

Antoine Ledent, Waleed Mustafa, Yunwen Lei, Marius Kloft

摘要:

我们展示了深度学习的泛化误差界,与现有技术相比有两个主要的改进:(1)我们的界除了对数因子外,对类的数量没有显式的依赖性。即使在权重矩阵的$L^2$范数方面制定界限时,这也是成立的,其中先前的界限至少表现出对类数的平方根依赖性。(2)我们采用经典的Rademacher分析DNN,将权重共享-一项具有基本理论重要性的任务,以前仅在非常严格的假设下尝试。在我们的结果中,每个卷积滤波器只对边界贡献一次,无论它被应用了多少次。提供了利用池化和稀疏连接的进一步改进。所提出的边界尺度作为参数矩阵的范数,而不是参数的数量。特别是,相反的边界基于参数计数,它们是渐近紧(对数因子)时的权重接近初始化,使它们适合作为一个基本成分的边界敏感的优化过程。我们还展示了如何适应最近的技术损失函数增强我们的情况,以取代光谱规范的经验类似物,同时保持我们的方法的优势。

英文摘要:

We show generalisation error bounds for deep learning with two main improvements over the state of the art. (1) Our bounds have no explicit dependence on the number of classes except for logarithmic factors. This holds even when formulating the bounds in terms of the $L^2$-norm of the weight matrices, where previous bounds exhibit at least a square-root dependence on the number of classes. (2) We adapt the classic Rademacher analysis of DNNs to incorporate weight sharing -- a task of fundamental theoretical importance which was previously attempted only under very restrictive assumptions. In our results, each convolutional filter contributes only once to the bound, regardless of how many times it is applied. Further improvements exploiting pooling and sparse connections are provided. The presented bounds scale as the norms of the parameter matrices, rather than the number of parameters. In particular, contrary to bounds based on parameter counting, they are asymptotically tight (up to log factors) when the weights approach initialisation, making them suitable as a basic ingredient in bounds sensitive to the optimisation procedure. We also show how to adapt the recent technique of loss function augmentation to our situation to replace spectral norms by empirical analogues whilst maintaining the advantages of our approach.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑