我们在当前使用随机梯度下降（SGD）及其变体训练深度网络的实践中发现了受抑制的数值不稳定性。我们表明，数值误差（量级为最小浮点位，因而也是深度网络训练中浮点算术所诱发的最极端或极限数值扰动）可能被显著放大，并导致显著的测试准确率方差（敏感性），其程度可与SGD随机性所导致的测试准确率方差相比。我们展示这很可能可追溯到优化动力学中受到抑制的不稳定性，即局限于若干迭代以及权重张量空间的局部区域。为此，我们提出一个利用偏微分方程（PDE）数值分析的理论框架，并分析卷积神经网络（CNN）的梯度下降PDE。我们表明，只有在学习率和权重衰减满足某些条件时，该PDE才是稳定的。我们表明，当这些条件被违反时，不稳定性并不会导致发散，而可能受到抑制。我们证明这是与CNN梯度下降相关的非线性PDE的结果：当离散化步长被过度驱动时，其局部线性化会发生变化，从而产生稳定化效应。我们将受抑制的不稳定性与最近发现的稳定边缘（Edge of Stability，EoS）现象联系起来；在该现象中，经典理论预测的稳定步长被超过，但损失仍继续优化并且依然收敛。由于受抑制的不稳定性发生在EoS处，我们的理论为EoS提供了新的见解和预测，特别是正则化的作用以及对网络复杂度的依赖性。
英文摘要：
We discover restrained numerical instabilities in current training practices of deep networks with stochastic gradient descent (SGD), and its variants. We show numerical error (on the order of the smallest floating point bit and thus the most extreme or limiting numerical perturbations induced from floating point arithmetic in training deep nets can be amplified significantly and result in significant test accuracy variance (sensitivities), comparable to the test accuracy variance due to stochasticity in SGD. We show how this is likely traced to instabilities of the optimization dynamics that are restrained, i.e., localized over iterations and regions of the weight tensor space. We do this by presenting a theoretical framework using numerical analysis of partial differential equations (PDE), and analyzing the gradient descent PDE of convolutional neural networks (CNNs). We show that it is stable only under certain conditions on the learning rate and weight decay. We show that rather than blowing up when the conditions are violated, the instability can be restrained. We show this is a consequence of the non-linear PDE associated with the gradient descent of the CNN, whose local linearization changes when over-driving the step size of the discretization, resulting in a stabilizing effect. We link restrained instabilities to the recently discovered Edge of Stability (EoS) phenomena, in which the stable step size predicted by classical theory is exceeded while continuing to optimize the loss and still converging. Because restrained instabilities occur at the EoS, our theory provides new insights and predictions about the EoS, in particular, the role of regularization and the dependence on the network complexity.
Di-BiLPS: Denoising induced Bidirectional Latent-PDE-Solver under Sparse Observations
School of Science, Harbin Institute of Technology, Shenzhen, China（哈尔滨工业大学深圳校区科学学院）；Department of Theoretical Physics（理论物理系）；Applied Mathematics, University of Cambridge, Cambridge, United Kingdom（应用数学系，剑桥大学，英国）
Center for Combustion Energy, Tsinghua University（清华大学燃烧能源中心）；College of AI, Tsinghua University（清华大学人工智能学院）；Department of Mechanical Engineering, University College London（伦敦大学学院机械工程系）
Purdue University（普渡大学）；Center for Computational and Applied Mathematics（计算与应用数学中心）；Department of Mathematics（数学系）
The extreme learning machine (ELM) method can yield highly accurate solutions to linear/nonlinear partial differential equations (PDEs), but requires the last hidden layer of the neural network to be wide to achieve a high accuracy. If the last hidden layer is narrow, the accuracy of the existing ELM method will be poor, irrespective of the rest of the network configuration. In this paper we present a modified ELM method, termed HLConcELM (hidden-layer concatenated ELM), to overcome the above drawback of the conventional ELM method. The HLConcELM method can produce highly accurate solutions to…
ShanghaiTech University（上海科技大学）；Imperial College London（帝国理工学院）；ETH Zürich（苏黎世联邦理工学院）；Tsinghua University（清华大学）；Noah’s Ark Lab, Huawei（华为诺亚方舟实验室）；University College London（伦敦大学学院）
Mainstream numerical Partial Differential Equation (PDE) solvers require discretizing the physical domain using a mesh. Mesh movement methods aim to improve the accuracy of the numerical solution by increasing mesh resolution where the solution is not well-resolved, whilst reducing unnecessary resolution elsewhere. However, mesh movement methods, such as the Monge-Ampere method, require the solution of auxiliary equations, which can be extremely expensive especially when the mesh is adapted frequently. In this paper, we propose to our best knowledge the first learning-based end-to-end mesh mov…
Shape Derivative-Informed Neural Operators with Application to Risk-Averse Shape Optimization
Oden Institute for Computational Engineering and Sciences, The University of Texas at Austin, Austin, Texas, USA（奥登计算工程与科学研究所，德克萨斯大学奥斯汀分校，奥斯汀，德克萨斯，美国）；School of Mathematical Sciences, Queensland University of Technology, Brisbane, Australia（数学科学学院，昆士兰科技大学，布里斯班，澳大利亚）；Centre for Data Science, Queensland University of Technology, Brisbane, Australia（数据科学中心，昆士兰科技大学，布里斯班，澳大利亚）；Department of Mathematics, The Ohio State University, Columbus, Ohio, USA（数学系，俄亥俄州立大学，哥伦布，俄亥俄，美国）；Department of Engineering Science and Biomedical Engineering, University of Auckland, Auckland, New Zealand（工程科学与生物医学工程系，奥克兰大学，奥克兰，新西兰）；Walker Department of Mechanical Engineering, The University of Texas at Austin, Austin, Texas, USA（沃克机械工程系，德克萨斯大学奥斯汀分校，奥斯汀，德克萨斯，美国）
As an emerging technology in deep learning, physics-informed neural networks (PINNs) have been widely used to solve various partial differential equations (PDEs) in engineering. However, PDEs based on practical considerations contain multiple physical quantities and complex initial boundary conditions, thus PINNs often returns incorrect results. Here we take heat transfer problem in multilayer fabrics as a typical example. It is coupled by multiple temperature fields with strong correlation, and the values of variables are extremely unbalanced among different dimensions. We clarify the potenti…
Finite Operator Learning: Bridging Neural Operators and Numerical Methods for Efficient Parametric Solution and Optimization of PDEs
Chair of Structural Analysis, Technical University of Munich（慕尼黑技术大学结构分析系）；Institute for Structural Analysis, Technische Universität Dresden（德累斯顿技术大学结构分析研究所）
Indian Institute of Technology Bombay（印度理工学院孟买分校）；Center for Machine Intelligence and Data Science (CMINDS), Indian Institute of Technology Bombay（印度理工学院孟买分校机器智能与数据科学中心（CMINDS））
We introduce a class of Sparse, Physics-based, and partially Interpretable Neural Networks (SPINN) for solving ordinary and partial differential equations (PDEs). By reinterpreting a traditional meshless representation of solutions of PDEs we develop a class of sparse neural network architectures that are partially interpretable. The SPINN model we propose here serves as a seamless bridge between two extreme modeling tools for PDEs, namely dense neural network based methods like Physics Informed Neural Networks (PINNs) and traditional mesh-free numerical methods, thereby providing a novel mean…
Learning Neural Constitutive Laws From Motion Observations for Generalizable PDE Dynamics
MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；MIT BCS（麻省理工学院脑与认知科学系）；Center for Brains, Minds and Machines（脑、心智与机器中心）；Tsinghua University（清华大学）；Shanghai Qi Zhi Institute（上海期智研究院）；UMass Amherst（马萨诸塞大学阿默斯特分校）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
Fast training of accurate physics-informed neural networks without gradient descent
Technical University of Munich（慕尼黑技术大学）；TUM Institute for Advanced Study（TUM高级研究学院）；Munich Center for Machine Learning（慕尼黑机器学习中心）；Wageningen University & Research（瓦赫宁根大学与研究中心）；KTH Royal Institute of Technology（皇家理工学院）；Munich Data Science Institute（慕尼黑数据科学研究所）
In recent years, deep learning has proven to be a viable methodology for surrogate modeling and uncertainty quantification for a vast number of physical systems. However, in their traditional form, such models can require a large amount of training data. This is of particular importance for various engineering and scientific applications where data may be extremely expensive to obtain. To overcome this shortcoming, physics-constrained deep learning provides a promising methodology as it only utilizes the governing equations. In this work, we propose a novel auto-regressive dense encoder-decode…
Indian Institute of Technology, Madras（印度理工学院马德拉斯分校）
There has been rapid progress recently on the application of deep networks to the solution of partial differential equations, collectively labelled as Physics Informed Neural Networks (PINNs). In this paper, we develop Physics Informed Extreme Learning Machine (PIELM), a rapid version of PINNs which can be applied to stationary and time dependent linear partial differential equations. We demonstrate that PIELM matches or exceeds the accuracy of PINNs on a range of problems. We also discuss the limitations of neural network based approaches, including our PIELM, in the solution of PDEs on large…
A Rapid Physics-Informed Machine Learning Framework Based on Extreme Learning Machine for Inverse Stefan Problems
School of Qilu Transportation, Shandong University（山东大学齐鲁交通学院）；School of Mechanical Engineering, Qilu University of Technology (Shandong Academy of Sciences)（齐鲁工业大学（山东省科学院）机械工程学院）
Physics Informed Neural Networks for design optimisation of diamond particle detectors for charged particle fast-tracking at high luminosity hadron colliders
Istituto Nazionale di Fisica Nucleare, Sezione di Firenze（意大利佛罗伦萨核物理研究所）；ICSC - Centro Nazionale di Ricerca in High Performance Computing, Big Data & Quantum Computing（高性能计算、大数据与量子计算国家研究中心）
