发表机构
Peking University; Massachusetts Institute of Technology; Tsinghua University（北京大学; 麻省理工学院; 清华大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究揭示注意力机制的非线性会产生损失的逆深度衰减，使大型语言模型可并行学习强弱谱方向，为深度缩放的来源提供了新解释。
AI 中文摘要
缩放定律描述了模型性能随数据集规模和参数数量呈幂律提升的现象，但其潜在机制尚未完全明晰。为解释参数数量缩放，现有理论假设模型深度呈幂律缩放。在线性注意力模型中，该缩放与幂律数据谱相关：由于无法选择性关注相关 token，这些模型会根据全局谱强度进行学习，较强方向先于较弱方向被学习。然而，大型语言模型可具有强非线性。本文表明，在所有测试的数据谱中，非线性注意力会产生损失的逆深度衰减。非线性使注意力能够选择性聚焦于相关 token，从而让强弱谱方向可并行学习。各层间的类似聚焦促使其与中心极限定理建立关联：各层间的共享误差设定了损失平台，而聚合过程将层间差异转化为随深度持续提升的增益。研究结果表明，深度缩放可能源于注意力机制的非线性，这使大型语言模型能够局部聚焦，并可能降低全局协方差结构的相关性。
英文摘要
Scaling laws describe power-law improvements in model performance with dataset size and parameter count, yet their underlying mechanisms are not fully understood. To explain the parameter count scaling, existing theory posits power-law scaling with model depth. In linear-attention models, this scaling is tied to a power-law data spectrum: unable to selectively attend to relevant tokens, these models learn according to global spectral strength, with stronger directions learned before weaker ones. Large language models, however, can be strongly nonlinear. Here, we show that nonlinear attention yields inverse-depth decay of loss across all tested data spectra. Nonlinearity enables attention to focus selectively on relevant tokens, allowing strong and weak spectral directions to be learned in parallel. Similar focusing across layers motivates a connection to the central limit theorem: shared error across layers sets the loss plateau, while aggregation turns layer-specific differences into continued gains with depth. Our findings suggest that depth scaling may arise from nonlinearity in attention, which allows large language models to focus locally and may make the global covariance structure less relevant.
Comments30 pages, 14 figures, 5 tables
