arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.26489cs.LG
收藏

奇异分布的稳定性:语言模型预训练两阶段动力学的谱视角

The Stability of Singular Distribution: A Spectral Perspective on the Two-Phase Dynamics of Language Model Pre-training

School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing, China(中国科学院大学先进交叉学科学院);State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China(中国科学院人工智能安全国家重点实验室);University of Chinese Academy of Sciences, Beijing, China(中国科学院大学);School of Mathematics, Southeast University, Nanjing, China(东南大学数学学院)

Hongtao Zhang, Wenjie Zhou, Chenxi Jia, Wei Chen, Xueqi Cheng

AI总结:

本文发现语言模型预训练中奇异值谱的早期稳定现象(SoSD),并证明该现象与慢下降阶段同步,通过理论分析揭示了权重范数增长导致SoSD阈值,从而限制后续损失下降速率。

摘要:

大型语言模型预训练通常表现出两阶段轨迹:快速的初始损失下降,随后是长时间的缓慢改善。我们识别出一个潜在的谱现象——奇异分布的稳定性(SoSD),其中迹归一化的奇异值谱早期就稳定下来,即使参数矩阵继续演化。我们证明,SoSD与慢下降阶段之间的同步在不同架构(GPT-2、LLaMA)和设置中广泛存在,包括各种调度(Step-wise、WSD、Cosine Decay)、权重衰减和优化器(AdamW、Muon)。通过分析一个简化的Transformer,我们证明权重范数的增长不可避免地会引发早期的SoSD阈值,之后损失下降速率在理论上受限于奇异分布的变化。我们进一步解释了WSD和Muon等策略通过调节SoSD尺度来影响预训练动态,从而为理解高效预训练动力学提供了谱视角。

英文摘要:

Large language model pre-training typically exhibits a two-phase trajectory: a fast initial loss drop followed by a prolonged slow improvement. We identify an underlying spectral phenomenon, Stability of Singular Distribution (SoSD), where the trace-normalized singular value spectrum stabilizes early, even as parameter matrices continue to evolve. We demonstrate that synchronization between SoSD and the slow-descent regime is widely observed across diverse architectures (GPT-2, LLaMA) and settings, including various schedules (Step-wise, WSD, Cosine Decay), weight decays, and optimizers (AdamW, Muon). By analyzing a simplified Transformer, we prove that growing weight norms inevitably precipitate an early SoSD threshold, after which the rate of loss decrease becomes theoretically bounded by the variation in the singular distribution. We further interpret strategies like WSD and Muon through their ability to modulate the SoSD scale, offering a spectral lens for understanding efficient pre-training dynamics.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑