arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.01045cs.LGcs.AIphysics.data-anstat.ML
收藏

叠加统一了幂律训练动态

Superposition unifies power-law training dynamics

Department of Physics, Massachusetts Institute of Technology, Cambridge, Massachusetts 02139, USA(麻省理工学院物理系);Department of Electrical and Computer Engineering, Princeton University, Princeton, New Jersey 08544, USA(普林斯顿大学电气与计算机工程系);Department of Mechanical Engineering, Massachusetts Institute of Technology, Cambridge, Massachusetts 02139, USA(麻省理工学院机械工程系)

Zixin Jessie Chen, Hao Chen, Yizhou Liu, Jeff Gore

AI总结:

叠加技术通过统一幂律动态,显著加速了训练过程,使训练速度提升十倍,适用于多种神经网络。

摘要:

我们通过教师-学生框架研究了特征叠加在幂律训练动态出现中的作用。我们首先推导了无叠加训练的解析理论,确立了幂律训练指数取决于输入数据统计和通道重要性。令人惊讶的是,我们发现叠加瓶颈会引发过渡到一个普遍的幂律指数约为1,与数据和通道统计无关。这种叠加的训练方式在时间上比无叠加的纯顺序学习快了十倍。我们的发现,即叠加导致数据无关的幂律指数快速训练,可能对采用叠加的广泛神经网络有重要影响,包括生产级大语言模型。

英文摘要:

We investigate the role of feature superposition in the emergence of power-law training dynamics using a teacher-student framework. We first derive an analytic theory for training without superposition, establishing that the power-law training exponent depends on both the input data statistics and channel importance. Remarkably, we discover that a superposition bottleneck induces a transition to a universal power-law exponent of $\sim 1$, independent of data and channel statistics. This one over time training with superposition represents an up to tenfold acceleration compared to the purely sequential learning that takes place in the absence of superposition. Our finding that superposition leads to rapid training with a data-independent power law exponent may have important implications for a wide range of neural networks that employ superposition, including production-scale large language models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑