arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2001.08361cs.LGstat.ML
收藏

神经语言模型的缩放定律

Scaling Laws for Neural Language Models

Johns Hopkins University(约翰斯·霍普金斯大学);OpenAI(开放人工智能研究中心)

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei

摘要:

我们研究经验缩放定律的语言模型性能的交叉熵损失。损失与模型大小、数据集大小和用于训练的计算量呈幂律关系,某些趋势跨越七个数量级以上。其他架构细节(如网络宽度或深度)在很大范围内具有最小的影响。简单的方程控制过拟合对模型/数据集大小的依赖性以及训练速度对模型大小的依赖性。这些关系使我们能够确定固定计算预算的最佳分配。更大的模型样本效率更高,因此最佳计算效率训练涉及在相对少量的数据上训练非常大的模型,并在收敛之前停止。

英文摘要:

We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude. Other architectural details such as network width or depth have minimal effects within a wide range. Simple equations govern the dependence of overfitting on model/dataset size and the dependence of training speed on model size. These relationships allow us to determine the optimal allocation of a fixed compute budget. Larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑