arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.16979cs.LGcond-mat.dis-nncs.AIstat.ML
收藏

用于分析大语言模型训练动态的可扩展损失景观曲率度量

A Scalable Measure of Loss Landscape Curvature for Analyzing the Training Dynamics of LLMs

Meta Superintelligence Labs(Meta超智能实验室);University of Maryland, College Park(马里兰大学哥伦比亚分校)

Dayal Singh Kalra, Jean-Christophe Gagnon-Audet, Andrey Gromov, Ishita Mediratta, Kelvin Niu, Alexander H Miller, Michael Shvartsman

AI总结:

本文提出了一种可扩展的损失景观曲率度量,用于分析大语言模型的训练动态,展示了大规模下的尖锐度现象,并指导数据混合策略。

摘要:

理解损失景观曲率演变对于分析神经网络的训练动态至关重要。最常研究的度量是Hessian尖锐度(λ_H^max)——损失Hessian的最大特征值——决定了局部训练稳定性,并在整个训练过程中与学习率相互作用。尽管其在分析训练动态中具有重要意义,但直接测量Hessian尖锐度对于大语言模型(LLMs)来说仍然具有很高的计算成本。我们分析了critical sharpness(λ_c),一种计算效率高的度量,仅需少于10次前向传递即可在给定更新方向Δθ的情况下完成。关键的是,这种度量能够很好地捕捉到已记录的Hessian尖锐度现象,包括逐步变尖和稳定性边缘。使用这种度量,我们首次在大规模范围内展示了这些尖锐度现象,达到70亿参数,涵盖OLMo-2模型的预训练和中期训练。我们进一步引入了相对critical sharpness(λ_c^{1→2}),它量化了在优化另一个损失景观时的曲率,以分析从预训练到微调的过渡,并指导数据混合策略。critical sharpness为从业者提供了一种实用工具,用于诊断曲率动态并指导大规模数据组成选择。更广泛地说,我们的工作表明,可扩展的曲率度量可以为大规模训练提供可操作的见解。

英文摘要:

Understanding the curvature evolution of the loss landscape is fundamental to analyzing the training dynamics of neural networks. The most commonly studied measure, Hessian sharpness ($λ_{\max}^H$) -- the largest eigenvalue of the loss Hessian -- determines local training stability and interacts with the learning rate throughout training. Despite its significance in analyzing training dynamics, direct measurement of Hessian sharpness remains prohibitive for Large Language Models (LLMs) due to high computational cost. We analyze $\textit{critical sharpness}$ ($λ_c$), a computationally efficient measure requiring fewer than $10$ forward passes given the update direction $Δ\mathbfθ$. Critically, this measure captures well-documented Hessian sharpness phenomena, including progressive sharpening and Edge of Stability. Using this measure, we provide the first demonstration of these sharpness phenomena at scale, up to $7$B parameters, spanning both pre-training and mid-training of OLMo-2 models. We further introduce $\textit{relative critical sharpness}$ ($λ_c^{1\to 2}$), which quantifies the curvature of one loss landscape while optimizing another, to analyze the transition from pre-training to fine-tuning and guide data mixing strategies. Critical sharpness provides practitioners with a practical tool for diagnosing curvature dynamics and informing data composition choices at scale. More broadly, our work shows that scalable curvature measures can provide actionable insights for large-scale training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑