The recent success of matrix optimizers (e.g., Muon) suggests that specific normalization of momentum, such as orthogonalization and row-wise normalization, benefits both the stability and acceleration of LLM training. Consequently, several recent studies have suggested that weights should also be normalized, leading to a Riemannian optimization problem. While such constrained training frameworks demonstrate superior performance, the effects of explicitly constraining weights, and their interaction with existing stabilization mechanisms, remain less understood. To bridge this gap, we study manifold constrained training dynamics through activation scales, rotational dynamics, and the update-to-weight ratio. We propose a Riemannian spectral steepest descent optimizer called MACRO, alongside a radius selection principle to serve as our testbed. Our analysis and numerical experiments reveal that RMSNorm and manifold constraints serve overlapping roles, and that weight decay can be completely eliminated when manifold constraints are applied. By controlling the update-to-weight ratio, constrained training significantly alleviates update cancellation, empirically demonstrating that MACRO is robust to low-precision computation and competitive with existing algorithms for standard LLM pre-training.
Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models
Department of Computer Science, Rice University, Houston, USA(莱斯大学计算机科学系,美国休斯顿);Department of Computer Science, University of North Carolina at Charlotte, Charlotte, USA(北卡罗来纳大学夏洛特分校计算机科学系,美国夏洛特);Department of Computer Science, Johns Hopkins University, Baltimore, USA(约翰霍普金斯大学计算机科学系,美国巴尔的摩);Department of Computer and Data Sciences, Case Western Reserve University(凯斯西储大学计算机与数据科学系)
Demystifying the Roles of LLM Layers in Retrieval, Knowledge, and Reasoning
Emory University(埃默里大学);University of Tuebingen(图宾根大学);University of Surrey(萨里大学);Max Planck Institute for Intelligent Systems(智能系统马克斯·普朗克研究所);ELLIS Institute Tübingen(图宾根ELLIS研究所)
Demystifying the Privacy-Utility Trade-off in LLM Interactions
Soochow University(苏州大学);Beihang University(北京航空航天大学);Suzhou City University(苏州城市学院);Shanghai Key Lab of Intelligent Information Processing(上海市智能信息处理重点实验室);Shanghai AI Lab(上海人工智能实验室)
Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoning
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学北京校区人工智能学院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);University College London, University of London(伦敦大学大学学院);Dalian University of Technology(大连理工大学)
Demystifying Issues, Causes and Solutions in LLM Open-Source Projects
School of Computer Science, Wuhan University(武汉大学计算机学院);School of Computer Science & Hubei Provincial Key Laboratory of Artificial Intelligence and Smart Learning, Central China Normal University(华中师范大学计算机学院);School of Computing Technologies, RMIT University(RMIT大学计算技术学院)
Demystifying amortized causal discovery with transformers
Institute of Science and Technology Austria(奥地利科学与技术研究所);Chan Zuckerberg Initiative(查纳·泽克伯格倡议);Mila - Quebec AI Institute and Université de Montréal(蒙特利尔AI研究所和蒙特利尔大学)