Algorithmic Foundations of Deep Learning: Complexity-Theoretic Rates and a Characterization of Universal Approximation
McMaster University & Vector Institute(麦克马斯特大学与向量研究所);Concordia University(康考迪亚大学);University of Tokyo(东京大学);McMaster University(麦克马斯特大学);University of Cambridge & University of Oxford(剑桥大学与牛津大学)
Anastasis Kratsios, Simone Brugiapaglia, Bum Jun Kim, Gregory Cousins, Haitz Sáez de Ocáriz Borde
Feedforward neural network (NN) expressivity is typically studied by emulating optimal basis-expansion schemes. While powerful, this perspective is incomplete: it primarily captures complexity through regularity, and therefore does not distinguish intuitively simple and complicated objects with comparable regularity, such as the square-root function and a typical Brownian path.
The guiding message is that neural networks should be viewed not only as flexible basis functions, but also as models of computation. If a function is computable by a real-valued circuit over a prescribed elementary gate language, then it can be computed to comparable accuracy by an NN with explicit depth, width, and non-zero-parameter bounds controlled by the depth, width, gate count, and gate structure. Thus, neural-network complexity is not governed by regularity alone, but also by algorithmic complexity. We then show that any definable NN model satisfying a natural parallelization condition, allowing possibly multivariate non-linearities such as attention or layer normalization, is a universal approximator if and only if it contains a non-affine nonlinearity.
The scope of our theory is illustrated by deducing universal approximation guarantees for continuous functions, minimax-optimal approximation guarantees for Besov classes, logarithmic-error complexity for holomorphic functions, and by showing that NNs can emulate numerical algorithms such as Newton-Raphson root finding and power iteration without architecture-specific arguments. Its precision is illustrated by shortest-path computation on $k$-vertex graphs: compiling the tropical dynamic-programming circuit yields NNs with O(log(1/ε)) non-zero parameters, exponentially improving in 1/ε over the generic $O(ε^{-c k^2})$ Lipschitz-approximation scale, for a constant c>0.
Statistical and Algorithmic Foundations of Reinforcement Learning
Department of Statistics and Data Science, Yale University(耶鲁大学统计与数据科学系);Department of Statistics and Data Science, University of Pennsylvania(宾夕法尼亚大学统计与数据科学系)
A Simplicity Bubble Problem in Formal-Theoretic Learning Systems
University of Campinas (UNICAMP)(坎皮纳斯大学);National Laboratory for Scientific Computing (LNCC)(国家科学计算实验室);Laboratoire de Recherche Scientifique (LABORES) for the Natural and Digital Sciences(自然与数字科学科学研究实验室(LABORES));The University of Cambridge(剑桥大学);Oxford Immune Algorithmics(牛津免疫算法公司);Karolinska Institute(卡罗林斯卡学院)