MSign: 通过稳定秩恢复防止大语言模型训练不稳定
MSign: An Optimizer Preventing Training Instability in Large Language Models via Stable Rank Restoration
AI总结:
MSign通过稳定秩恢复机制防止大语言模型训练不稳定,有效减少训练失败并降低计算开销
摘要:
大语言模型(LLM)预训练中仍存在训练不稳定的问题,通常表现为突然的梯度爆炸,浪费大量计算资源。我们研究了一个通过μP扩展的500万参数NanoGPT模型中的训练失败现象,发现崩溃前存在两个关键现象:(1)权重矩阵稳定秩(平方Frobenius范数与平方谱范数的比值)迅速下降;(2)相邻层雅可比矩阵之间的对齐性逐渐增加。我们理论证明这两个条件共同导致网络深度下的指数级梯度范数增长。为打破这种不稳定性机制,我们提出MSign,一种新的优化器,通过定期应用矩阵符号操作来恢复稳定秩。在500万到30亿参数模型上的实验表明,MSign能够有效防止训练失败,计算开销低于7.0%。
英文摘要:
Training instability remains a critical challenge in large language model (LLM) pretraining, often manifesting as sudden gradient explosions that waste significant computational resources. We study training failures in a 5M-parameter NanoGPT model scaled via $μ$P, identifying two key phenomena preceding collapse: (1) rapid decline in weight matrix stable rank (ratio of squared Frobenius norm to squared spectral norm), and (2) increasing alignment between adjacent layer Jacobians. We prove theoretically that these two conditions jointly cause exponential gradient norm growth with network depth. To break this instability mechanism, we propose MSign, a new optimizer that periodically applies matrix sign operations to restore stable rank. Experiments on models from 5M to 3B parameters demonstrate that MSign effectively prevents training failures with a computational overhead of less than 7.0%.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。