We analyze feature learning in infinite-width neural networks trained with gradient flow through a self-consistent dynamical field theory. We construct a collection of deterministic dynamical order parameters which are inner-product kernels for hidden unit activations and gradients in each layer at pairs of time points, providing a reduced description of network activity through training. These kernel order parameters collectively define the hidden layer activation distribution, the evolution of the neural tangent kernel, and consequently output predictions. We show that the field theory derivation recovers the recursive stochastic process of infinite-width feature learning networks obtained from Yang and Hu (2021) with Tensor Programs . For deep linear networks, these kernels satisfy a set of algebraic matrix equations. For nonlinear networks, we provide an alternating sampling procedure to self-consistently solve for the kernel order parameters. We provide comparisons of the self-consistent solution to various approximation schemes including the static NTK approximation, gradient independence assumption, and leading order perturbation theory, showing that each of these approximations can break down in regimes where general self-consistent solutions still provide an accurate description. Lastly, we provide experiments in more realistic settings which demonstrate that the loss and kernel dynamics of CNNs at fixed feature learning strength is preserved across different widths on a CIFAR classification task.
Learning Dynamical Systems from Data: A Simple Cross-Validation Perspective, Part V: Sparse Kernel Flows for 132 Chaotic Dynamical Systems
Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Imperial College London（帝国理工学院）；Zhejiang Zhelixin Credit Investigation Co., Ltd.（浙江浙里信征信有限公司）；Caltech（加州理工学院）；Johns Hopkins University（约翰斯·霍普金斯大学）
Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networks
John Paulson School of Engineering and Applied Sciences（约翰·保尔森工程与应用科学学院）；Center for Brain Science（脑科学中心）；Kempner Institute for the Study of Natural & Artificial Intelligence（自然与人工智能研究肯普纳研究所）；Harvard University（哈佛大学）
Fast Approximation and Estimation Bounds of Kernel Quadrature for Infinitely Wide Models
RIKEN AIP（日本理化学研究所革新智能综合研究中心）
无限宽模型是特征图的加权积分$\int φ(x,v) d μ(v)$。该模型擅长处理无限数量的特征，因此已被用于深度学习的理论研究。核求积是一种基于核的数值积分方案，用于快速逼近期望值$\int f(x) d p(x)$。在这项研究中，关于重量$μ$作为一个有符号（或复杂/向量值）分布的参数，我们开发的一般核求积（GKQ）的参数分布。所提出的方法可以实现一个快速的近似速度$O(e^{-p})$与参数数$p$，这是比传统的巴伦的速度，和一个快速的估计速度$\widetilde{O}(1/n)$与样本量$n$。因此，我们得到了一个新的基于范数的复杂性度量的无限宽模型。由于GKQ隐式地进行经验风险最小化，因此我们可以理解复杂性度量也反映了梯度学习设置中的泛化性能。
Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networks
Department of Physics, Harvard University（哈佛大学物理系）；Center for Brain Science, Harvard University（哈佛大学脑科学中心）；John A. Paulson School of Engineering and Applied Sciences, Harvard University（哈佛大学约翰·A·保尔森工程与应用科学学院）
Forecasting and Granger Modelling with Non-linear Dynamical Dependencies
Geneva School of Business Administration, HES-SO University of Applied Sciences of Western Switzerland（日内瓦商学院，瑞士西部应用科学与艺术大学）；University of Geneva（日内瓦大学）
Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks
John A. Paulson School of Engineering and Applied Sciences, Harvard University（哈佛大学约翰·A·保尔森工程与应用科学学院）；Center for Brain Science, Harvard University（哈佛大学脑科学中心）
Learning the gravitational force law and other analytic functions
Google Research（谷歌研究院）
Large neural network models have been successful in learning functions of importance in many branches of science, including physics, chemistry and biology. Recent theoretical work has shown explicit learning bounds for wide networks and kernel methods on some simple classes of functions, but not on more complex functions which arise in practice. We extend these techniques to provide learning bounds for analytic functions on the sphere for any kernel method or equivalent infinitely-wide network with the corresponding activation function trained with SGD. We show that a wide, one-hidden layer Re…
Kernel Methods for Stochastic Dynamical Systems with Application to Koopman Eigenfunctions: Feynman-Kac Representations and RKHS Approximation
Department of Computing and Mathematical Sciences, Caltech（加州理工学院计算与数学科学系）；The Alan Turing Institute（艾伦·图灵研究所）；Department of Mechanical Engineering, Clemson University（克莱姆森大学机械工程系）
