Universal Approximation Power of Deep Residual Neural Networks via Nonlinear Control Theory
University of California at Los Angeles（加州大学洛杉矶分校）
Paulo Tabuada, Bahman Gharesifard
备注：
Sejun Park and Geonho Hwang brought to our atention a mistake in the proof of Theorem 5.1. This mistake is corrected in this version with the consequence of increasing the number of neurons per layer from n+1 to 2n+1
In this paper, we explain the universal approximation capabilities of deep residual neural networks through geometric nonlinear control. Inspired by recent work establishing links between residual networks and control systems, we provide a general sufficient condition for a residual network to have the power of universal approximation by asking the activation function, or one of its derivatives, to satisfy a quadratic differential equation. Many activation functions used in practice satisfy this assumption, exactly or approximately, and we show this property to be sufficient for an adequately deep neural network with $n+1$ neurons per layer to approximate arbitrarily well, on a compact set and with respect to the supremum norm, any continuous function from $\mathbb{R}^n$ to $\mathbb{R}^n$. We further show this result to hold for very simple architectures for which the weights only need to assume two values. The first key technical contribution consists of relating the universal approximation problem to controllability of an ensemble of control systems corresponding to a residual network and to leverage classical Lie algebraic techniques to characterize controllability. The second technical contribution is to identify monotonicity as the bridge between controllability of finite ensembles and uniform approximability on compact sets.
We demonstrate that in residual neural networks (ResNets) dynamical isometry is achievable irrespectively of the activation function used. We do that by deriving, with the help of Free Probability and Random Matrix Theories, a universal formula for the spectral density of the input-output Jacobian at initialization, in the large network width and depth limit. The resulting singular value spectrum depends on a single parameter, which we calculate for a variety of popular activation functions, by analyzing the signal propagation in the artificial neural network. We corroborate our results with n…
Minimum Block Width for Universal Approximation by Residual Neural Networks with Inner Width One
School of Mathematics and Statistics, Huazhong University of Science and Technology（华中科技大学数学与统计学院）；Hubei Key Laboratory of Engineering Modeling and Scientific Computing, Huazhong University of Science and Technology（湖北省工程建模与科学计算重点实验室）
Universal Approximation Constraints of Narrow ResNets: The Tunnel Effect
Technical University of Munich（慕尼黑工业大学）；Munich Data Science Institute (MDSI)（慕尼黑数据科学研究所）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；TU Wien（维也纳工业大学）
Normalizing flows are a class of flexible deep generative models that offer easy likelihood computation. Despite their empirical success, there is little theoretical understanding of their expressiveness. In this work, we study residual flows, a class of normalizing flows composed of Lipschitz residual blocks. We prove residual flows are universal approximators in maximum mean discrepancy. We provide upper bounds on the number of residual blocks to achieve approximation under different assumptions.
Fudan University（复旦大学）；The University of Sydney（悉尼大学）
The performance of GNNs degrades as they become deeper due to the over-smoothing. Among all the attempts to prevent over-smoothing, residual connection is one of the promising methods due to its simplicity. However, recent studies have shown that GNNs with residual connections only slightly slow down the degeneration. The reason why residual connections fail in GNNs is still unknown. In this paper, we investigate the forward and backward behavior of GNNs with residual connections from a novel path decomposition perspective. We find that the recursive aggregation of the median length paths from…
Massachusetts Institute of Technology Lincoln Laboratory Supercomputing Center（麻省理工学院林肯实验室超级计算中心）
A Multigrid Full Approximation Storage algorithm for solving Deep Residual Networks is developed to enable neural network parallelized layer-wise training and concurrent computational kernel execution on GPUs. This work demonstrates a 10.2x speedup over traditional layer-wise model parallelism techniques using the same number of compute units.
Acoustic echo cannot be entirely removed by linear adaptive filters due to the nonlinear relationship between the echo and far-end signal. Usually a post processing module is required to further suppress the echo. In this paper, we propose a residual echo suppression method based on the modification of fully convolutional time-domain audio separation network (Conv-TasNet). Both the residual signal of the linear acoustic echo cancellation system, and the output of the adaptive filter are adopted to form multiple streams for the Conv-TasNet, resulting in more effective echo suppression while kee…
We regard pre-trained residual networks (ResNets) as nonlinear systems and use linearization, a common method used in the qualitative analysis of nonlinear systems, to understand the behavior of the networks under small perturbations of the input images. We work with ResNet-56 and ResNet-110 trained on the CIFAR-10 data set. We linearize these networks at the level of residual units and network stages, and the singular value decomposition is used in the stability analysis of these components. It is found that most of the singular values of the linearizations of residual units are 1 and, in spi…
Broad stochastic configuration residual learning system for norm-convergent universal approximation
School of Control and Computer Engineering, North China Electric Power University（控制与计算机工程学院，华北电力大学）；School of Mathematics and Physics, North China Electric Power University（数学与物理学院，华北电力大学）；School of Intelligent Systems Engineering, Shenzhen Campus of Sun Yat-sen University（智能系统工程学院，中山大学深圳校区）
Residual-loss Anomaly Analysis of Physics-Informed Neural Networks: An Inverse Method for Change-point Detection in Nonlinear Dynamical Systems with Regime Switching
School of Mathematics and Statistics（数学与统计学学院）；Huazhong University of Science and Technology（华中科技大学）；Northwestern Polytechnical University（西北工业大学）
Eindhoven University of Technology（埃因霍温理工大学）；Cooperatieve Rabobank U.A.（拉博银行）
Effectively detecting anomalous nodes in attributed networks is crucial for the success of many real-world applications such as fraud and intrusion detection. Existing approaches have difficulties with three major issues: sparsity and nonlinearity capturing, residual modeling, and network smoothing. We propose Residual Graph Convolutional Network (ResGCN), an attention-based deep residual modeling approach that can tackle these issues: modeling the attributed networks with GCN allows to capture the sparsity and nonlinearity; utilizing a deep neural network allows to directly learn residual fro…
Princeton University（普林斯顿大学）；Beijing Institute of Big Data Research（北京大数据研究院）
Optimal a priori estimates are derived for the population risk, also known as the generalization error, of a regularized residual network model. An important part of the regularized model is the usage of a new path norm, called the weighted path norm, as the regularization term. The weighted path norm treats the skip connections and the nonlinearities differently so that paths with more nonlinearities are regularized by larger weights. The error estimates are a priori in the sense that the estimates depend only on the target function, not on the parameters obtained in the training process. The…
COGNOS: Universal Enhancement for Time Series Anomaly Detection via Constrained Gaussian-Noise Optimization and Smoothing
School of Information Science and Technology, Beijing University of Technology, China（信息科学与技术学院，北京理工大学，中国）；Beijing Key Laboratory of Computational Intelligence and Intelligent System, Beijing University of Technology, China（北京计算智能与智能系统重点实验室，北京理工大学，中国）
Residual-Based Error Corrector Operator to Enhance Accuracy and Reliability of Neural Operator Surrogates of Nonlinear Variational Boundary-Value Problems
Oden Institute for Computational Engineering and Sciences, The University of Texas at Austin（德克萨斯大学奥斯汀分校奥登计算工程与科学研究所）
Euler Institute, Università della Svizzera Italiana（瑞士意大利语区大学欧拉研究所）
We train deep residual networks with a stochastic variant of the nonlinear multigrid method MG/OPT. To build the multilevel hierarchy, we use the dynamical systems viewpoint specific to residual networks. We report significant speed-ups and additional robustness for training MNIST on deep residual networks. Our numerical experiments also indicate that multilevel training can be used as a pruning technique, as many of the auxiliary networks have accuracies comparable to the original network.
Adaptive Randomized Neural Networks for PDEs: Structure-Aware Feature Redistribution and Residual-Driven Block Enrichment
School of Mathematics and Statistics, Xi’an Jiaotong University（西安交通大学数学与统计学院）；School of Mathematics and Statistics & State Key Laboratory of Multiphase Flow in Power Engineering, Xi’an Jiaotong University（西安交通大学数学与统计学院）
