Nondeterminism in neural network optimization produces uncertainty in performance, making small improvements difficult to discern from run-to-run variability. While uncertainty can be reduced by training multiple model copies, doing so is time-consuming, costly, and harms reproducibility. In this work, we establish an experimental protocol for understanding the effect of optimization nondeterminism on model diversity, allowing us to isolate the effects of a variety of sources of nondeterminism. Surprisingly, we find that all sources of nondeterminism have similar effects on measures of model diversity. To explain this intriguing fact, we identify the instability of model training, taken as an end-to-end procedure, as the key determinant. We show that even one-bit changes in initial parameters result in models converging to vastly different values. Last, we propose two approaches for reducing the effects of instability on run-to-run variability.
Taming the Instability: A Robust Second-Order Optimizer for Federated Learning over Non-IID Data
Key Laboratory of Collaborative Intelligence Systems of Ministry of Education, Xidian University（西安电子科技大学教育部协同智能系统重点实验室）；Centre for Frontier AI Research, Institute of High Performance Computing, Agency for Science, Technology and Research（新加坡科技研究局高性能计算研究所前沿人工智能研究中心）；State Key Laboratory of ISN, School of Cyber Engineering, Xidian University（西安电子科技大学网络与信息安全学院综合业务网理论及关键技术国家重点实验室）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；Department of Computing Technologies, Swinburne University of Technology（斯威本科技大学计算技术系）
Theoretical Analysis of Sparse Optimization with Reparameterization, Weight Decay, and Adaptive Learning Rate
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China（人工智能安全国家重点实验室，计算技术研究所，中国科学院，北京，中国）；School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing, China（中国科学院大学计算机科学与技术学院，北京，中国）；Beijing Academy of Artificial Intelligence (BAAI), Beijing, China（北京人工智能研究院（BAAI），北京，中国）；IIIS, Tsinghua University, Beijing, China（清华大学人工智能院，北京，中国）；School of Statistics and Management, Shanghai University of Finance and Economics, Shanghai, China（上海财经大学统计与管理学院，上海，中国）；Institute of Data Science and Statistics, Shanghai University of Finance and Economics, Shanghai, China（上海财经大学数据科学与统计研究所，上海，中国）
Training instability in deep learning follows low-dimensional dynamical principles
China Mobile Research Institute（中国移动研究院）；China Mobile GBA Innovation Institute（中国移动广州创新研究院）；Institute of Microelectronics, Chinese Academy of Sciences（中国科学院微电子研究所）
Accelerating PDE-Constrained Optimization by the Derivative of Neural Operators
Bosch (China) Invest Ltd., Shanghai, China；Dept. of Comp. Sci. \& Techn., Institute for AI, BNRist Center, Tsinghua-Bosch Joint ML Center, Tsinghua University
Clipping-Free Policy Optimization for Large Language Models
KUIS AI Center, Koç University, Istanbul, Türkiye（Koç大学）；Koç University, Istanbul, Türkiye（Koç大学）；University of California, Berkeley, CA, USA（加州大学伯克利分校）
Proximal Policy Optimization with Mixed Distributed Training
Shanghai Institute for Advanced Communication and Data Science（上海先进通信与数据科学研究院）；School of Computer Engineering and Science, Shanghai University（上海大学计算机工程与科学学院）
SIPO: Stabilized and Improved Preference Optimization for Aligning Diffusion Models
Shanghai Science and Intelligence Institute, Shanghai, China（上海科学与智能研究所）；Fudan University, Shanghai, China（复旦大学）；Australian Institute for Machine Learning, The University of Adelaide（澳大利亚机器学习研究所，阿德莱德大学）
On the Stability Properties and the Optimization Landscape of Training Problems with Squared Loss for Neural Networks and General Nonlinear Conic Approximation Schemes
