arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.26705cs.LGcs.AIcs.LOcs.NAmath.NA
收藏

深度学习算法基础:复杂度理论速率与通用逼近的表征

Algorithmic Foundations of Deep Learning: Complexity-Theoretic Rates and a Characterization of Universal Approximation

McMaster University & Vector Institute(麦克马斯特大学与向量研究所);Concordia University(康考迪亚大学);University of Tokyo(东京大学);McMaster University(麦克马斯特大学);University of Cambridge & University of Oxford(剑桥大学与牛津大学)

Anastasis Kratsios, Simone Brugiapaglia, Bum Jun Kim, Gregory Cousins, Haitz Sáez de Ocáriz Borde

AI总结:

本文从计算模型角度研究神经网络,证明若函数可由基本门电路计算,则神经网络能以可控深度和宽度逼近,并给出通用逼近的充要条件。

摘要:

前馈神经网络(NN)的表达性通常通过模拟最优基展开方案来研究。虽然这种方法强大,但视角不完整:它主要通过正则性来捕捉复杂度,因此无法区分具有可比正则性的直观简单与复杂对象,例如平方根函数和典型布朗路径。核心信息是,神经网络不仅应被视为灵活的基函数,还应被视为计算模型。如果一个函数可由预设基本门语言上的实值电路计算,那么它可以通过具有显式深度、宽度和非零参数界限(由深度、宽度、门数量和门结构控制)的神经网络以可比精度计算。因此,神经网络复杂度不仅由正则性决定,还受算法复杂度影响。我们进一步证明,任何满足自然并行化条件(允许使用注意力或层归一化等多变量非线性)的可定义神经网络模型,当且仅当其包含非仿射非线性时才是通用逼近器。通过推导连续函数的通用逼近保证、Besov类的最小最大最优逼近保证、全纯函数的对数误差复杂度,以及证明神经网络无需特定架构参数即可模拟牛顿-拉夫森求根和幂迭代等数值算法,说明了我们理论的范围。其精确性通过$k$顶点图上的最短路径计算说明:编译热带动态规划电路得到具有$O(\log(1/{\epsilon}))$个非零参数的神经网络,在$1/{\epsilon}$上指数级优于通用的$O({\epsilon}^{-c k^2})$ Lipschitz逼近尺度(其中$c>0$为常数)。

英文摘要:

Feedforward neural network (NN) expressivity is typically studied by emulating optimal basis-expansion schemes. While powerful, this perspective is incomplete: it primarily captures complexity through regularity, and therefore does not distinguish intuitively simple and complicated objects with comparable regularity, such as the square-root function and a typical Brownian path. The guiding message is that neural networks should be viewed not only as flexible basis functions, but also as models of computation. If a function is computable by a real-valued circuit over a prescribed elementary gate language, then it can be computed to comparable accuracy by an NN with explicit depth, width, and non-zero-parameter bounds controlled by the depth, width, gate count, and gate structure. Thus, neural-network complexity is not governed by regularity alone, but also by algorithmic complexity. We then show that any definable NN model satisfying a natural parallelization condition, allowing possibly multivariate non-linearities such as attention or layer normalization, is a universal approximator if and only if it contains a non-affine nonlinearity. The scope of our theory is illustrated by deducing universal approximation guarantees for continuous functions, minimax-optimal approximation guarantees for Besov classes, logarithmic-error complexity for holomorphic functions, and by showing that NNs can emulate numerical algorithms such as Newton-Raphson root finding and power iteration without architecture-specific arguments. Its precision is illustrated by shortest-path computation on $k$-vertex graphs: compiling the tropical dynamic-programming circuit yields NNs with O(log(1/ε)) non-zero parameters, exponentially improving in 1/ε over the generic $O(ε^{-c k^2})$ Lipschitz-approximation scale, for a constant c>0.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑