arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1910.05697cs.LGstat.ML
收藏

基于近似描述长度的神经网络泛化界

Generalization Bounds for Neural Networks via Approximate Description Length

The Hebrew University(希伯来大学);Google Research(谷歌研究院)

Amit Daniely, Elad Granot

摘要:

我们研究了网络的样本复杂性,其权重的大小有界。特别地,我们考虑类\[ H=\left\{W_t\circρ\circ \ldots\circρ\circ W_{1} :W_1,\ldots,W_{t-1}\in M_{d, d}, W_t\in M_{1,d}\right\} \],其中每个$W_i$的谱范数由$O(1)$界定,Frobenius范数由$R$界定,并且$ρ$是S形函数$\frac{e^x}{1+e^x}$或平滑的ReLU函数$ \ln (1+e^x)$。我们表明,对于任何深度的$t$,如果输入是在$[-1,1]^d$,$H$的样本复杂度是$\tilde O\left(\frac{dR^2}{ε^2}\right)$。该界限对于对数因子是最佳的,并且大大改进了$\tilde O\left(\frac{d^2R^2}{ε^2}\right)$的先前技术状态。我们还证明了,如果不考虑$W_i$的大小,我们考虑$W_i - W_i^0$的大小,其中$W_i^0$是一些参考矩阵,与$O(1)$的谱范数,这个界限仍然有效。通过将$W_i^0$作为训练过程开始时的矩阵,我们得到了在许多典型的参数制度中,在参数数量上是次线性的样本复杂度界限。为了建立我们的结果,我们开发了一种新的技术来分析样本的复杂性家庭$H$的预测。我们首先定义了一个新的概念的随机近似描述的功能$f:X\to\mathbb{R}^d$。然后,我们表明,如果有一种方法来近似描述功能的类$H$使用$d$位,然后$d/ε^2$的例子足以保证一致收敛。也就是说,类中所有函数的经验损失是$ε$ -接近真实损失。最后,我们开发了一套工具,用于计算近似描述长度的函数类,可以作为一个组合的线性函数类和非线性函数。

英文摘要:

We investigate the sample complexity of networks with bounds on the magnitude of its weights. In particular, we consider the class \[ H=\left\{W_t\circρ\circ \ldots\circρ\circ W_{1} :W_1,\ldots,W_{t-1}\in M_{d, d}, W_t\in M_{1,d}\right\} \] where the spectral norm of each $W_i$ is bounded by $O(1)$, the Frobenius norm is bounded by $R$, and $ρ$ is the sigmoid function $\frac{e^x}{1+e^x}$ or the smoothened ReLU function $ \ln (1+e^x)$. We show that for any depth $t$, if the inputs are in $[-1,1]^d$, the sample complexity of $H$ is $\tilde O\left(\frac{dR^2}{ε^2}\right)$. This bound is optimal up to log-factors, and substantially improves over the previous state of the art of $\tilde O\left(\frac{d^2R^2}{ε^2}\right)$. We furthermore show that this bound remains valid if instead of considering the magnitude of the $W_i$'s, we consider the magnitude of $W_i - W_i^0$, where $W_i^0$ are some reference matrices, with spectral norm of $O(1)$. By taking the $W_i^0$ to be the matrices at the onset of the training process, we get sample complexity bounds that are sub-linear in the number of parameters, in many typical regimes of parameters. To establish our results we develop a new technique to analyze the sample complexity of families $H$ of predictors. We start by defining a new notion of a randomized approximate description of functions $f:X\to\mathbb{R}^d$. We then show that if there is a way to approximately describe functions in a class $H$ using $d$ bits, then $d/ε^2$ examples suffices to guarantee uniform convergence. Namely, that the empirical loss of all the functions in the class is $ε$-close to the true loss. Finally, we develop a set of tools for calculating the approximate description length of classes of functions that can be presented as a composition of linear function classes and non-linear functions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑