On the linearity of large non-linear models: when and why the tangent kernel is constant
Department of Computer Science, The Ohio State University（俄亥俄州立大学计算机科学系）；Halicioğlu Data Science Institute, University of California, San Diego（加利福尼亚大学圣迭戈分校哈利西奥卢数据科学学院）
Chaoyue Liu, Libin Zhu, Mikhail Belkin
备注：
accepted as Spotlight in NeurIPS 2020; made correction to proof
The goal of this work is to shed light on the remarkable phenomenon of transition to linearity of certain neural networks as their width approaches infinity. We show that the transition to linearity of the model and, equivalently, constancy of the (neural) tangent kernel (NTK) result from the scaling properties of the norm of the Hessian matrix of the network as a function of the network width. We present a general framework for understanding the constancy of the tangent kernel via Hessian scaling applicable to the standard classes of neural networks. Our analysis provides a new perspective on the phenomenon of constant tangent kernel, which is different from the widely accepted "lazy training". Furthermore, we show that the transition to linearity is not a general property of wide neural networks and does not hold when the last layer of the network is non-linear. It is also not necessary for successful optimization by gradient descent.
Ohio State University（俄亥俄州立大学）；UC, San Diego（加州大学圣地亚哥分校）
Wide neural networks with linear output layer have been shown to be near-linear, and to have near-constant neural tangent kernel (NTK), in a region containing the optimization path of gradient descent. These findings seem counter-intuitive since in general neural networks are highly complex models. Why does a linear structure emerge when the networks become wide? In this work, we provide a new perspective on this "transition to linearity" by considering a neural network as an assembly model recursively built from a set of sub-models corresponding to individual neurons. In this view, we show th…
Physics-informed neural networks (PINNs) have lately received great attention thanks to their flexibility in tackling a wide range of forward and inverse problems involving partial differential equations. However, despite their noticeable empirical success, little is known about how such constrained neural networks behave during their training via gradient descent. More importantly, even less is known about why such models sometimes fail to train at all. In this work, we aim to investigate these questions through the lens of the Neural Tangent Kernel (NTK); a kernel that captures the behavior …
Peking University（北京大学）；Georgia Institute of Technology（佐治亚理工学院）
Deep residual networks (ResNets) have demonstrated better generalization performance than deep feedforward networks (FFNets). However, the theory behind such a phenomenon is still largely unknown. This paper studies this fundamental problem in deep learning from a so-called "neural tangent kernel" perspective. Specifically, we first show that under proper conditions, as the width goes to infinity, training deep ResNets can be viewed as learning reproducing kernel functions with some kernel function. We then compare the kernel of deep ResNets with that of deep FFNets and discover that the class…
Optimization dependent generalization bound for ReLU networks based on sensitivity in the tangent bundle
Institute for Computer Science and Control (SZTAKI)（计算机科学与控制研究所（SZTAKI））；Eötvös Loránd University（厄特沃什·罗兰大学）；CNRS CRIStAL（法国国家科学研究中心信号与自动化信息处理实验室）；Université de Lille（里尔大学）；Széchenyi István University（塞切尼·伊什特万大学）
A Unified Generalization Framework for Model Merging: Trade-offs, Non-Linearity, and Scaling Laws
National University of Defense Technology（国防科技大学）；Wuhan University（武汉大学）；School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University（中山大学信息科学与技术学院深圳校区）；Hebei University of Technology（河北工业大学）
The University of Chicago（芝加哥大学）；Chicago Quantum Exchange（芝加哥量子交易所）；qBraid Co.（qBraid公司）；HRL Laboratories, LLC（HRL实验室有限责任公司）；Argonne National Laboratory（阿贡国家实验室）
A fundamental bottleneck in utilising complex machine learning systems for critical applications has been not knowing why they do and what they do, thus preventing the development of any crucial safety protocols. To date, no method exist that can provide full insight into the granularity of the neural network's decision process. In the past, saliency maps were an early attempt at resolving this problem through sensitivity calculations, whereby dimensions of a data point are selected based on how sensitive the output of the system is to them. However, the success of saliency maps has been at be…
Unveiling the Mechanism of Continuous Representation Full-Waveform Inversion: A Wave Based Neural Tangent Kernel Framework
School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an, Shaanxi, China（西安交通大学数学与统计学学院）；Macao Institute of Systems Engineering, Macau University of Science and Technology, Taipa, Macao（澳门系统工程研究院，澳门科技大学，路环，澳门）
When and Why Adversarial Training Improves PINNs: A Neural Tangent Kernel Perspective
School of Mathematics and Statistics, Beijing Institute of Technology, China（北京理工大学数学与统计学院，中国）；School of Professional Education and Executive Development The Hong Kong Polytechnic University, China（香港理工大学专业教育学院及管理发展学院，中国）；Department of Computer Science & UCL AI Centre, University College London, UK（伦敦大学学院计算机科学系及UCL人工智能中心，英国）
Establishing a theoretical analysis that explains why deep learning can outperform shallow learning such as kernel methods is one of the biggest issues in the deep learning literature. Towards answering this question, we evaluate excess risk of a deep learning estimator trained by a noisy gradient descent with ridge regularization on a mildly overparameterized neural network, and discuss its superiority to a class of linear estimators that includes neural tangent kernel approach, random feature model, other kernel methods, $k$-NN estimator and so on. We consider a teacher-student regression mo…
Deep neural networks can empirically perform efficient hierarchical learning, in which the layers learn useful representations of the data. However, how they make use of the intermediate representations are not explained by recent theories that relate them to "shallow learners" such as kernels. In this work, we demonstrate that intermediate neural representations add more flexibility to neural networks and can be advantageous over raw inputs. We consider a fixed, randomly initialized neural network as a representation function fed into another trainable network. When the trainable network is t…
Microsoft Research AI（微软研究院人工智能部门）；Stanford University（斯坦福大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）
The fundamental learning theory behind neural networks remains largely open. What classes of functions can neural networks actually learn? Why doesn't the trained network overfit when it is overparameterized? In this work, we prove that overparameterized neural networks can learn some notable concept classes, including two and three-layer networks with fewer parameters and smooth activations. Moreover, the learning can be simply done by SGD (stochastic gradient descent) or its variants in polynomial time using polynomially many samples. The sample complexity can also be almost independent of t…
Natural Gradient Descent (NGD) helps to accelerate the convergence of gradient descent dynamics, but it requires approximations in large-scale deep neural networks because of its high computational cost. Empirical studies have confirmed that some NGD methods with approximate Fisher information converge sufficiently fast in practice. Nevertheless, it remains unclear from the theoretical perspective why and under what conditions such heuristic approximations work well. In this work, we reveal that, under specific conditions, NGD with approximate Fisher information achieves the same fast converge…
With the motive of training all the parameters of a neural network, we study why and when one can achieve this by iteratively creating, training, and combining randomly selected subnetworks. Such scenarios have either implicitly or explicitly emerged in the recent literature: see e.g., the Dropout family of regularization techniques, or some distributed ML training protocols that reduce communication/computation complexities, such as the Independent Subnet Training protocol. While these methods are studied empirically and utilized in practice, they often enjoy partial or no theoretical support…
Recent work in the literature has shown experimentally that one can use the lower layers of a trained convolutional neural network (CNN) to model natural textures. More interestingly, it has also been experimentally shown that only one layer with random filters can also model textures although with less variability. In this paper we ask the question as to why one layer CNNs with random filters are so effective in generating textures? We theoretically show that one layer convolutional architectures (without a non-linearity) paired with the an energy function used in previous literature, can in …
École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）；Imperial College London（帝国理工学院）
At initialization, artificial neural networks (ANNs) are equivalent to Gaussian processes in the infinite-width limit, thus connecting them to kernel methods. We prove that the evolution of an ANN during training can also be described by a kernel: during gradient descent on the parameters of an ANN, the network function $f_θ$ (which maps input vectors to output vectors) follows the kernel gradient of the functional cost (which is convex, in contrast to the parameter cost) w.r.t. a new kernel: the Neural Tangent Kernel (NTK). This kernel is central to describe the generalization features of ANN…
Explaining Bayesian Optimization by Shapley Values Facilitates Human-AI Collaboration
Ludwig-Maximilians-Universität (LMU) Munich（慕尼黑大学）；John A. Paulson Harvard School of Engineering and Applied Sciences, Harvard University（哈佛大学约翰·保尔森工程与应用科学学院）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Wyss Institute for Biologically Inspired Engineering, Harvard University（哈佛大学怀斯生物启发工程研究所）
Microsoft Research AI（微软研究院AI）；Stanford University（斯坦福大学）；Princeton University（普林斯顿大学）；UT-Austin（德克萨斯大学奥斯汀分校）；University of Washington（华盛顿大学）；Harvard University（哈佛大学）
Deep neural networks (DNNs) have demonstrated dominating performance in many fields; since AlexNet, networks used in practice are going wider and deeper. On the theoretical side, a long line of works has been focusing on training neural networks with one hidden layer. The theory of multi-layer networks remains largely unsettled. In this work, we prove why stochastic gradient descent (SGD) can find $\textit{global minima}$ on the training objective of DNNs in $\textit{polynomial time}$. We only make two assumptions: the inputs are non-degenerate and the network is over-parameterized. The latter…
The Quantum Path Kernel: a Generalized Quantum Neural Tangent Kernel for Deep Quantum Machine Learning
University of Verona（维罗纳大学）；European Organization for Nuclear Research (CERN)（欧洲核子研究组织（CERN））；University of Gdansk（格但斯克大学）；Middlesex University（密德萨斯大学）
Quantized neural networks have drawn a lot of attention as they reduce the space and computational complexity during the inference. Moreover, there has been folklore that quantization acts as an implicit regularizer and thus can improve the generalizability of neural networks, yet no existing work formalizes this interesting folklore. In this paper, we take the binary weights in a neural network as random variables under stochastic rounding, and study the distribution propagation over different layers in the neural network. We propose a quasi neural network to approximate the distribution prop…
The Devil is in the Condition Numbers: Why is GLU Better than non-GLU Structure?
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China（人工智能安全国家重点实验室，计算技术研究所，中国科学院，北京100190，中国）；School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 101408, China（中国科学院大学计算机科学与技术学院，北京101408，中国）；Beijing Academy of Artificial Intelligence (BAAI), Beijing, China（北京人工智能研究院（BAAI），北京，中国）
Offline reinforcement learning (RL) leverages previously collected data for policy optimization without any further active exploration. Despite the recent interest in this problem, its theoretical results in neural network function approximation settings remain elusive. In this paper, we study the statistical theory of offline RL with deep ReLU network function approximation. In particular, we establish the sample complexity of $n = \tilde{\mathcal{O}}( H^{4 + 4 \frac{d}α} κ_μ^{1 + \frac{d}α} ε^{-2 - 2\frac{d}α} )$ for offline RL with deep ReLU networks, where $κ_μ$ is a measure of distributio…
Department of Electrical Engineering, Tsinghua University（清华大学电子工程系）
The developments of deep neural networks (DNN) in recent years have ushered a brand new era of artificial intelligence. DNNs are proved to be excellent in solving very complex problems, e.g., visual recognition and text understanding, to the extent of competing with or even surpassing people. Despite inspiring and encouraging success of DNNs, thorough theoretical analyses still lack to unravel the mystery of their magics. The design of DNN structure is dominated by empirical results in terms of network depth, number of neurons and activations. A few of remarkable works published recently in an…
Physics-informed neural network (PINN) algorithms have shown promising results in solving a wide range of problems involving partial differential equations (PDEs). However, they often fail to converge to desirable solutions when the target function contains high-frequency features, due to a phenomenon known as spectral bias. In the present work, we exploit neural tangent kernels (NTKs) to investigate the training dynamics of PINNs evolving under stochastic gradient descent with momentum (SGDM). This demonstrates SGDM significantly reduces the effect of spectral bias. We have also examined why …
Deep convolutional neural networks have led to breakthrough results in numerous practical machine learning tasks such as classification of images in the ImageNet data set, control-policy-learning to play Atari games or the board game Go, and image captioning. Many of these applications first perform feature extraction and then feed the results thereof into a trainable classifier. The mathematical analysis of deep convolutional neural networks for feature extraction was initiated by Mallat, 2012. Specifically, Mallat considered so-called scattering networks based on a wavelet transform followed…
Rethinking Influence Functions of Neural Networks in the Over-parameterized Regime
Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；School of Mathematical Sciences, University of Chinese Academy of Sciences（中国科学院大学数学科学学院）
