从自然语言统计中推导神经缩放定律
Deriving Neural Scaling Laws from the statistics of natural language
AI总结：
研究从自然语言统计推导神经缩放定律，分离出语言关键统计属性预测神经缩放指数，给出基于这些统计量的简单公式，与实验测量定律匹配。
摘要：
尽管实验性神经缩放定律极大地推动了大规模机器学习的实证进展，但现有理论无法定量预测在任何自然语言数据集上训练的任何现代语言模型的这些重要定律的指数。我们在数据受限缩放定律的情况下提供了第一个这样的理论。我们分离出语言的两个关键统计属性，仅这些属性就能预测神经缩放指数：(i) 成对令牌相关性随令牌对之间时间间隔的衰减，以及 (ii) 下一个令牌条件熵随条件上下文长度的衰减。我们进一步根据这些统计量推导出一个简单的公式，该公式从第一原理预测数据受限的神经缩放指数，无需任何自由参数或合成数据模型。我们的理论与通过在两个定性不同的基准TinyStories和WikiText上从头开始训练GPT-2和LLaMA风格模型获得的实验测量神经缩放定律表现出显著匹配。
英文摘要：
Despite the fact that experimental neural scaling laws have substantially guided empirical progress in large-scale machine learning, no existing theory can quantitatively predict the exponents of these important laws for any modern LLM trained on any natural language dataset. We provide the first such theory in the case of data-limited scaling laws. We isolate two key statistical properties of language that alone can predict neural scaling exponents: (i) the decay of pairwise token correlations with time separation between token pairs, and (ii) the decay of the next-token conditional entropy with the length of the conditioning context. We further derive a simple formula in terms of these statistics that predicts data-limited neural scaling exponents from first principles without any free parameters or synthetic data models. Our theory exhibits a remarkable match with experimentally measured neural scaling laws obtained from training GPT-2 and LLaMA style models from scratch on two qualitatively different benchmarks, TinyStories and WikiText.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
