In LLM training, normalization layers and optimizers are typically treated as independent design choices. In a 3x2 factorial at 1B parameters and 1000 training steps, we show this assumption can fail: Dynamic Erf (Derf; Chen & Liu, 2025) suffers a large negative interaction with Muon (Jordan, 2024), with its gap to RMSNorm growing from +0.31 nats under AdamW to +0.97 under Muon, approximately three times larger. Dynamic Tanh (DyT; Zhu et al., 2025), included as a bounded-normalizer control, shows no such penalty. Our evidence points to two failure modes of erf under Muon's faster spectral-norm growth: saturation (lossy compression) and scale blindness (discarding activation magnitude). An EMA-blend that reintroduces running scale estimates recovers ~84% of the gap. Separately, reducing Derf's alpha from its published default (0.5 to 0.3) recovers ~80% by keeping erf in its near-linear regime, where it approximately preserves relative scale; this setting is not the published default of Chen & Liu (2025). Using Derf's published default alpha with Muon incurs a 0.66-nat interaction penalty without producing NaNs or divergence, making the failure easy to miss in short pilot runs.
Conda: Column-Normalized Adam for Training Large Language Models Faster
State Key Lab of General AI, School of Intelligence Science and Technology, Peking University(通用人工智能国家重点实验室,智能科学与技术学院,北京大学);Singapore Management University(新加坡国立大学);Nankai University(南开大学);Beijing Normal University(北京师范大学);ByteDance Seed(字节跳动种子);Beijing University of Posts and Telecommunications(北京邮电大学);Institute for Artificial Intelligence, Peking University(人工智能研究院,北京大学);Pazhou Laboratory (Huangpu), Guangzhou, Guangdong, China(黄埔实验室(广州),广东,中国)
AdamS: Momentum Itself Can Be A Normalizer for LLM Pretraining and Post-training
Wangxuan Institute of Computer Technology, Peking University(北京大学王宣计算机技术研究所);University of Science and Technology of China(中国科学技术大学);State Key Laboratory of General Artificial Intelligence(通用人工智能国家重点实验室)
Memory-Efficient LLM Pretraining via Minimalist Optimizer Design
Department of Electrical and Computer Engineering, University of Minnesota, USA(电气与计算机工程系,明尼苏达大学,美国);School of Mathematics and Statistics, University of Sydney, Australia(数学与统计学学院,悉尼大学,澳大利亚)
Normalize-Then-Precondition: A Hierarchical Approach to Marginal Scale and Interaction Geometry for LLM Training
Gaoling School of Artificial Intelligence(高瓴人工智能学院);Renmin University of China(中国人民大学);School of Statistics and Management(统计与管理学院);Shanghai University of Finance and Economics(上海财经大学)
Lightweight Geometric Adaptation for Training Physics-Informed Neural Networks
Department of Computational Applied Mathematics and Operations Research(计算应用数学与运筹学部门);School of Data Science(数据科学学院);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))
IQuest Research(IQuest研究公司);Peking University(北京大学);Sun Yat-sen University(中山大学);Shenzhen University of Advanced Technology(深圳先进技术大学);Shanghai University of Finance and Economics(上海财经大学)