arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.12446cs.LGcs.AImath.STstat.MLstat.TH
收藏

神经网络在低复杂度数据上实现泛化

Neural Networks Generalize on Low Complexity Data

Stanford University(斯坦福大学)

Sourav Chatterjee, Timothy Sudijono

AI总结:

本文提出基于最小描述长度的神经网络在低复杂度数据上实现泛化,并通过素数检测实验验证其有效性。

摘要:

我们证明了具有ReLU激活函数的前馈神经网络在适当定义的低复杂度数据上能够实现泛化。给定由简单编程语言生成的i.i.d.数据,最小描述长度(MDL)前馈神经网络在插值数据时能够以高概率实现泛化。我们定义了这种简单编程语言,并提出了此类网络的描述长度概念。我们提供了几个基本计算任务的例子,如检查自然数的素数性。在素数检测方面,我们的定理表明,假设我们从1到N均匀随机抽取n个数作为样本。对于每个数x_i,令y_i=1如果x_i是素数,否则为0。然后,插值MDL网络可以以概率1-O((ln N)/n)准确回答一个新抽取的1到N之间的数是否为素数。需要注意的是,该网络并非专门设计用于检测素数;最小描述学习发现了一个能够实现此功能的网络。还讨论了对噪声数据的扩展,表明MDL神经网络插值器可以展示出受控的过拟合现象。

英文摘要:

We show that feedforward neural networks with ReLU activation generalize on low complexity data, suitably defined. Given i.i.d.~data generated from a simple programming language, the minimum description length (MDL) feedforward neural network which interpolates the data generalizes with high probability. We define this simple programming language, along with a notion of description length of such networks. We provide several examples on basic computational tasks, such as checking primality of a natural number. For primality testing, our theorem shows the following and more. Suppose that we draw an i.i.d.~sample of $n$ numbers uniformly at random from $1$ to $N$. For each number $x_i$, let $y_i = 1$ if $x_i$ is a prime and $0$ if it is not. Then, the interpolating MDL network accurately answers, with probability $1- O((\ln N)/n)$, whether a newly drawn number between $1$ and $N$ is a prime or not. Note that the network is not designed to detect primes; minimum description learning discovers a network which does so. Extensions to noisy data are also discussed, suggesting that MDL neural network interpolators can demonstrate tempered overfitting.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑