我们在当前使用随机梯度下降（SGD）及其变体训练深度网络的实践中发现了受抑制的数值不稳定性。我们表明，数值误差（量级为最小浮点位，因而也是深度网络训练中浮点算术所诱发的最极端或极限数值扰动）可能被显著放大，并导致显著的测试准确率方差（敏感性），其程度可与SGD随机性所导致的测试准确率方差相比。我们展示这很可能可追溯到优化动力学中受到抑制的不稳定性，即局限于若干迭代以及权重张量空间的局部区域。为此，我们提出一个利用偏微分方程（PDE）数值分析的理论框架，并分析卷积神经网络（CNN）的梯度下降PDE。我们表明，只有在学习率和权重衰减满足某些条件时，该PDE才是稳定的。我们表明，当这些条件被违反时，不稳定性并不会导致发散，而可能受到抑制。我们证明这是与CNN梯度下降相关的非线性PDE的结果：当离散化步长被过度驱动时，其局部线性化会发生变化，从而产生稳定化效应。我们将受抑制的不稳定性与最近发现的稳定边缘（Edge of Stability，EoS）现象联系起来；在该现象中，经典理论预测的稳定步长被超过，但损失仍继续优化并且依然收敛。由于受抑制的不稳定性发生在EoS处，我们的理论为EoS提供了新的见解和预测，特别是正则化的作用以及对网络复杂度的依赖性。
英文摘要：
We discover restrained numerical instabilities in current training practices of deep networks with stochastic gradient descent (SGD), and its variants. We show numerical error (on the order of the smallest floating point bit and thus the most extreme or limiting numerical perturbations induced from floating point arithmetic in training deep nets can be amplified significantly and result in significant test accuracy variance (sensitivities), comparable to the test accuracy variance due to stochasticity in SGD. We show how this is likely traced to instabilities of the optimization dynamics that are restrained, i.e., localized over iterations and regions of the weight tensor space. We do this by presenting a theoretical framework using numerical analysis of partial differential equations (PDE), and analyzing the gradient descent PDE of convolutional neural networks (CNNs). We show that it is stable only under certain conditions on the learning rate and weight decay. We show that rather than blowing up when the conditions are violated, the instability can be restrained. We show this is a consequence of the non-linear PDE associated with the gradient descent of the CNN, whose local linearization changes when over-driving the step size of the discretization, resulting in a stabilizing effect. We link restrained instabilities to the recently discovered Edge of Stability (EoS) phenomena, in which the stable step size predicted by classical theory is exceeded while continuing to optimize the loss and still converging. Because restrained instabilities occur at the EoS, our theory provides new insights and predictions about the EoS, in particular, the role of regularization and the dependence on the network complexity.
Di-BiLPS: Denoising induced Bidirectional Latent-PDE-Solver under Sparse Observations
School of Science, Harbin Institute of Technology, Shenzhen, China（哈尔滨工业大学深圳校区科学学院）；Department of Theoretical Physics（理论物理系）；Applied Mathematics, University of Cambridge, Cambridge, United Kingdom（应用数学系，剑桥大学，英国）
Center for Combustion Energy, Tsinghua University（清华大学燃烧能源中心）；College of AI, Tsinghua University（清华大学人工智能学院）；Department of Mechanical Engineering, University College London（伦敦大学学院机械工程系）
Shape Derivative-Informed Neural Operators with Application to Risk-Averse Shape Optimization
Oden Institute for Computational Engineering and Sciences, The University of Texas at Austin, Austin, Texas, USA（奥登计算工程与科学研究所，德克萨斯大学奥斯汀分校，奥斯汀，德克萨斯，美国）；School of Mathematical Sciences, Queensland University of Technology, Brisbane, Australia（数学科学学院，昆士兰科技大学，布里斯班，澳大利亚）；Centre for Data Science, Queensland University of Technology, Brisbane, Australia（数据科学中心，昆士兰科技大学，布里斯班，澳大利亚）；Department of Mathematics, The Ohio State University, Columbus, Ohio, USA（数学系，俄亥俄州立大学，哥伦布，俄亥俄，美国）；Department of Engineering Science and Biomedical Engineering, University of Auckland, Auckland, New Zealand（工程科学与生物医学工程系，奥克兰大学，奥克兰，新西兰）；Walker Department of Mechanical Engineering, The University of Texas at Austin, Austin, Texas, USA（沃克机械工程系，德克萨斯大学奥斯汀分校，奥斯汀，德克萨斯，美国）
Finite Operator Learning: Bridging Neural Operators and Numerical Methods for Efficient Parametric Solution and Optimization of PDEs
Chair of Structural Analysis, Technical University of Munich（慕尼黑技术大学结构分析系）；Institute for Structural Analysis, Technische Universität Dresden（德累斯顿技术大学结构分析研究所）
SPINN: Sparse, Physics-based, and partially Interpretable Neural Networks for PDEs
Indian Institute of Technology Bombay（印度理工学院孟买分校）；Center for Machine Intelligence and Data Science (CMINDS), Indian Institute of Technology Bombay（印度理工学院孟买分校机器智能与数据科学中心（CMINDS））
Learning Neural Constitutive Laws From Motion Observations for Generalizable PDE Dynamics
MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；MIT BCS（麻省理工学院脑与认知科学系）；Center for Brains, Minds and Machines（脑、心智与机器中心）；Tsinghua University（清华大学）；Shanghai Qi Zhi Institute（上海期智研究院）；UMass Amherst（马萨诸塞大学阿默斯特分校）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
Fast training of accurate physics-informed neural networks without gradient descent
Technical University of Munich（慕尼黑技术大学）；TUM Institute for Advanced Study（TUM高级研究学院）；Munich Center for Machine Learning（慕尼黑机器学习中心）；Wageningen University & Research（瓦赫宁根大学与研究中心）；KTH Royal Institute of Technology（皇家理工学院）；Munich Data Science Institute（慕尼黑数据科学研究所）
A Rapid Physics-Informed Machine Learning Framework Based on Extreme Learning Machine for Inverse Stefan Problems
School of Qilu Transportation, Shandong University（山东大学齐鲁交通学院）；School of Mechanical Engineering, Qilu University of Technology (Shandong Academy of Sciences)（齐鲁工业大学（山东省科学院）机械工程学院）
Physics Informed Neural Networks for design optimisation of diamond particle detectors for charged particle fast-tracking at high luminosity hadron colliders
Istituto Nazionale di Fisica Nucleare, Sezione di Firenze（意大利佛罗伦萨核物理研究所）；ICSC - Centro Nazionale di Ricerca in High Performance Computing, Big Data & Quantum Computing（高性能计算、大数据与量子计算国家研究中心）
