Understanding the curvature evolution of the loss landscape is fundamental to analyzing the training dynamics of neural networks. The most commonly studied measure, Hessian sharpness ($λ_{\max}^H$) -- the largest eigenvalue of the loss Hessian -- determines local training stability and interacts with the learning rate throughout training. Despite its significance in analyzing training dynamics, direct measurement of Hessian sharpness remains prohibitive for Large Language Models (LLMs) due to high computational cost. We analyze $\textit{critical sharpness}$ ($λ_c$), a computationally efficient measure requiring fewer than $10$ forward passes given the update direction $Δ\mathbfθ$. Critically, this measure captures well-documented Hessian sharpness phenomena, including progressive sharpening and Edge of Stability. Using this measure, we provide the first demonstration of these sharpness phenomena at scale, up to $7$B parameters, spanning both pre-training and mid-training of OLMo-2 models. We further introduce $\textit{relative critical sharpness}$ ($λ_c^{1\to 2}$), which quantifies the curvature of one loss landscape while optimizing another, to analyze the transition from pre-training to fine-tuning and guide data mixing strategies. Critical sharpness provides practitioners with a practical tool for diagnosing curvature dynamics and informing data composition choices at scale. More broadly, our work shows that scalable curvature measures can provide actionable insights for large-scale training.
Evolving Safety Landscape of Multi-modal Large Language Models: A Survey of Emerging Threats and Safeguards
University of Alabama at Birmingham(阿拉巴马大学伯明翰分校);NVIDIA(英伟达公司);Penn State University(宾夕法尼亚州立大学);Columbia University(哥伦比亚大学);University of Missouri-Kansas City(密苏里大学堪萨斯分校);Florida State University(佛罗里达州立大学);Auburn University(奥本大学)
Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes
Peking University(北京大学);Beijing Institute of Big Data Research(北京大数据研究院);Center for Data Science, Peking University(北京大学数据科学中心);BICMR, Peking University(北京大学北京国际数学研究中心);Princeton University(普林斯顿大学)
What shapes the loss landscape of self-supervised learning?
The University of Tokyo(东京大学);Physics & Informatics Laboratories, NTT Research, Inc.(NTT研究公司物理与信息学实验室);Harvard University(哈佛大学);University of Michigan(密歇根大学);RIKEN Center for Emergent Matter Science (CEMS)(理化学研究所涌现物质科学中心)
Navigating the landscape of multimodal AI in medicine: a scoping review on technical challenges and clinical applications
Radboud University Medical Center(拉德堡德大学医学中心);Oncode Institute(Oncode研究所);Polytechnic University of Turin(都灵理工大学);University of Turin(都灵大学);Erasmus University Medical Centre(伊拉斯姆斯大学医学中心)
On the Optimization Landscape of Dynamic Output Feedback: A Case Study for Linear Quadratic Regulator
National University of Singapore(新加坡国立大学);University of Science and Technology Beijing(北京科技大学);Tsinghua University(清华大学);University of California San Diego(加利福尼亚大学圣迭戈分校)