arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.01225stat.MLcs.LGmath.STstat.TH
收藏

面向现代网络的路径范数工具箱:推论、前景与挑战

A path-norm toolkit for modern networks: consequences, promises and challenges

Univ Lyon(里昂大学);EnsL(里昂高等师范学院);UCBL(里昂第一大学);CNRS(法国国家科学研究中心);Inria(法国国家信息与自动化研究所);LIP(里昂计算机科学实验室)

Antoine Gonon, Nicolas Brisebarre, Elisa Riccietti, Rémi Gribonval

AI总结:

本文提出首个支持带偏置、跳跃连接及次序统计量操作的一般DAG ReLU网络的路径范数工具箱,推导更通用且更尖锐的泛化界,并在ImageNet的ResNet上进行数值检验。

摘要:

本工作介绍了首个围绕路径范数构建的工具箱,它通过提取次序统计量,完整涵盖了带偏置、跳跃连接以及任意操作(包括最大池化、GroupSort等)的一般DAG ReLU网络。该工具箱尤其使我们能够为现代神经网络建立泛化界;这些界不仅是适用范围最广的基于路径范数的泛化界,而且能够恢复或超越此类界中已知最尖锐的结果。这些扩展路径范数还享有路径范数的通常优势:易于计算、在网络对称性下保持不变,并且与另一种最常用的复杂度度量——算子范数乘积相比,在分层全连接网络上具有更高的尖锐性。该工具箱的多功能性及其易于实现的特点,使我们能够通过在ImageNet上对ResNet数值评估已知最尖锐的界,来挑战基于路径范数的泛化界的具体前景。

英文摘要:

This work introduces the first toolkit around path-norms that fully encompasses general DAG ReLU networks with biases, skip connections and any operation based on the extraction of order statistics: max pooling, GroupSort etc. This toolkit notably allows us to establish generalization bounds for modern neural networks that are not only the most widely applicable path-norm based ones, but also recover or beat the sharpest known bounds of this type. These extended path-norms further enjoy the usual benefits of path-norms: ease of computation, invariance under the symmetries of the network, and improved sharpness on layered fully-connected networks compared to the product of operator norms, another complexity measure most commonly used. The versatility of the toolkit and its ease of implementation allow us to challenge the concrete promises of path-norm-based generalization bounds, by numerically evaluating the sharpest known bounds for ResNets on ImageNet.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑