arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.22681cs.LG
收藏

通过平坦方向动力学增强加速大语言模型预训练

Accelerating LLM Pre-Training through Flat-Direction Dynamics Enhancement

Peking University(北京大学);Zhejiang Lab, Zhejiang(浙江实验室)

Shuchen Zhu, Rizhen Hu, Mingze Wang, Mou Sun, Xue Wang, Kun Yuan, Zaiwen Wen

AI总结:

LITE通过增强平坦方向动力学提升大语言模型预训练效率

摘要:

大语言模型的预训练需要巨大的计算资源,使优化器的效率变得至关重要。优化景观高度各向异性,损失减少主要由平坦方向上的进展驱动。尽管基于矩阵的优化器如Muon和SOAP利用细粒度曲率信息来超越AdamW,但它们的更新倾向于各向同性——在平坦方向上相对保守,在尖锐方向上可能激进。为了解决这一限制,我们首先建立了一个统一的黎曼微分方程(ODE)框架,阐明了常见自适应算法如何协同工作:预处理程序诱导了黎曼几何,以缓解病态条件,而动量则作为黎曼阻尼项促进收敛。受这些见解的启发,我们提出了LITE,一种通用的加速策略,通过在平坦轨迹上应用更大的Hessian阻尼系数和学习率来增强训练动态。广泛的实验表明,LITE在多种架构(密集型、MoE)、参数规模(1.3亿至13亿)、数据集(C4、Pile)和学习率调度(余弦、暖启动-稳定衰减)上显著加速了Muon和SOAP。理论分析证实,LITE在各向异性景观中加速了平坦方向的收敛,提供了一种高效大语言模型预训练的系统方法。代码可在https://github.com/SHUCHENZHU/LITE上获得。

英文摘要:

Pre-training Large Language Models requires immense computational resources, making optimizer efficiency essential. The optimization landscape is highly anisotropic, with loss reduction driven predominantly by progress along flat directions. While matrix-based optimizers such as Muon and SOAP use fine-grained curvature information to outperform AdamW, their updates tend toward isotropy, which is relatively conservative along flat directions yet potentially aggressive along sharp ones. To address this limitation, we first develop a unified Riemannian Ordinary Differential Equation (ODE) framework that offers a continuous-time interpretation of adaptive algorithms: the preconditioner induces a Riemannian geometry that mitigates ill-conditioning, while momentum serves as a Riemannian damping term that promotes convergence. Guided by these insights, we propose LITE, a generalized acceleration strategy that enhances training dynamics by applying larger Hessian damping coefficients and learning rates along flat trajectories. Extensive experiments demonstrate that LITE significantly accelerates both Muon and SOAP across diverse architectures (Dense, MoE), parameter scales (130M to 1.3B), datasets (C4, Pile), and learning-rate schedules (cosine, wsd). Theoretical analysis confirms that LITE facilitates faster convergence along flat directions in anisotropic landscapes, providing a principled approach to efficient LLM pre-training. The code is available at https://github.com/SHUCHENZHU/LITE.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑