arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.03685cs.LGcs.AIstat.ML
收藏

学习尖峰分布中的普适三分之一时间缩放

Universal One-third Time Scaling in Learning Peaked Distributions

University of California, Berkeley(加州大学伯克利分校)

Yizhou Liu, Ziming Liu, Cengiz Pehlevan, Jeff Gore

AI总结:

本文通过理论分析和实验验证,揭示了使用softmax和交叉熵学习尖峰分布时,损失和梯度呈幂律衰减,导致损失时间缩放指数为1/3的普适瓶颈,为神经缩放现象提供了机理解释。

摘要:

训练大型语言模型(LLM)计算成本高昂,部分原因是损失呈现缓慢的幂律收敛,其起源仍有争议。通过对玩具模型的系统分析和LLM的经验评估,我们表明这种行为本质上源于softmax和交叉熵的使用。当学习尖峰概率分布(例如下一个词元分布)时,这些组件普遍产生幂律衰减的损失和梯度,与许多微观细节无关,从而形成基本的优化瓶颈。这最终导致损失的时间缩放服从幂律,普适指数为$1/3$。我们的结果为观察到的神经缩放提供了机理解释,并提出了改进LLM训练效率的新方向。

英文摘要:

Training large language models (LLMs) is computationally expensive, partly because the loss exhibits slow power-law convergence whose origin remains debatable. Through systematic analysis of toy models and empirical evaluation of LLMs, we show that this behavior can arise intrinsically from the use of softmax and cross-entropy. When learning peaked probability distributions, e.g., next-token distributions, these components generically yield power-law vanishing losses and gradients, regardless of many microscopic details, creating a fundamental optimization bottleneck. This ultimately leads to power-law time scaling of the loss with a universal exponent of $1/3$. Our results provide a mechanistic explanation for observed neural scaling and suggest new directions for improving LLM training efficiency.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑