arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2004.10802cs.LGstat.ML
收藏

一种基于数据流形维数的神经缩放定律

A Neural Scaling Law from the Dimension of the Data Manifold

Johns Hopkins University(约翰斯·霍普金斯大学)

Utkarsh Sharma, Jared Kaplan

摘要:

当数据丰富时,经过良好训练的神经网络所实现的损失在网络参数$N$的数量上按幂律$L \propto N^{-α}$缩放。这种经验性的缩放定律适用于各种各样的数据模态,并且可以在许多数量级上持续存在。如果神经模型有效地只对内在维度为$d$的数据流形执行回归,则可以解释缩放定律。这个简单的理论预测了交叉熵和均方误差损失的标度指数$α\approx 4/d$。我们通过独立测量教师/学生框架中的内在维数和标度指数来证实理论,在这个框架中,我们可以通过拨打随机教师网络的属性来研究各种$d$和$α$。我们还在几个数据集上使用CNN图像分类器和GPT类型的语言模型测试了该理论。

英文摘要:

When data is plentiful, the loss achieved by well-trained neural networks scales as a power-law $L \propto N^{-α}$ in the number of network parameters $N$. This empirical scaling law holds for a wide variety of data modalities, and may persist over many orders of magnitude. The scaling law can be explained if neural models are effectively just performing regression on a data manifold of intrinsic dimension $d$. This simple theory predicts that the scaling exponents $α\approx 4/d$ for cross-entropy and mean-squared error losses. We confirm the theory by independently measuring the intrinsic dimension and the scaling exponents in a teacher/student framework, where we can study a variety of $d$ and $α$ by dialing the properties of random teacher networks. We also test the theory with CNN image classifiers on several datasets and with GPT-type language models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑