叠加统一了幂律训练动态
Superposition unifies power-law training dynamics
AI总结:
叠加技术通过统一幂律动态,显著加速了训练过程,使训练速度提升十倍,适用于多种神经网络。
摘要:
我们通过教师-学生框架研究了特征叠加在幂律训练动态出现中的作用。我们首先推导了无叠加训练的解析理论,确立了幂律训练指数取决于输入数据统计和通道重要性。令人惊讶的是,我们发现叠加瓶颈会引发过渡到一个普遍的幂律指数约为1,与数据和通道统计无关。这种叠加的训练方式在时间上比无叠加的纯顺序学习快了十倍。我们的发现,即叠加导致数据无关的幂律指数快速训练,可能对采用叠加的广泛神经网络有重要影响,包括生产级大语言模型。
英文摘要:
We investigate the role of feature superposition in the emergence of power-law training dynamics using a teacher-student framework. We first derive an analytic theory for training without superposition, establishing that the power-law training exponent depends on both the input data statistics and channel importance. Remarkably, we discover that a superposition bottleneck induces a transition to a universal power-law exponent of $\sim 1$, independent of data and channel statistics. This one over time training with superposition represents an up to tenfold acceleration compared to the purely sequential learning that takes place in the absence of superposition. Our finding that superposition leads to rapid training with a data-independent power law exponent may have important implications for a wide range of neural networks that employ superposition, including production-scale large language models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。