具有有界参数的神经网络的数值逼近容量：极限是否存在，又如何衡量？
Numerical Approximation Capacity of Neural Networks with Bounded Parameters: Do Limits Exist, and How Can They Be Measured?
AI总结：
该研究探讨了参数有界时神经网络的数值逼近极限，证明了有界参数空间下DNNs仅能由有限维向量空间逼近，并引入ε外测度和NSdim量化该极限，为网络宽度、深度及正则化等提供新见解。
摘要：
万能逼近定理指出，在具有合适的激活函数且参数可自由选择或训练的情况下，神经网络在理论上可以拥有无限的逼近能力。然而，当这些神经参数（尤其是非线性权重和偏置）有界时，会出现一种更具实用性的场景。这引发了我们提出问题：当参数在实际中受到有界限制时，神经网络的逼近能力是保持万能的，还是存在极限？如果存在极限，又该如何衡量？我们的理论研究指出，尽管万能逼近在理论上是可行的，但在实际数值场景中，具有任意解析激活函数（如 Tanh 和 Sigmoid）的深度神经网络（DNNs）在有界非线性参数空间（NP 空间）下，无论在连续还是离散意义上，都只能由有限维向量空间来逼近。基于此项研究，我们引入了 ε 外测度和数值张成维数（NSdim）的概念，从理论和实践两方面量化了一类网络的逼近能力极限。此外，借鉴我们新的理论研究并采用全新的视角，我们致力于探究反向传播神经网络与随机参数网络（如极限学习机（ELM））在有限和无限宽度下的关系。我们还旨在为正则化、宽度与深度的权衡、参数空间、宽度冗余、凝聚以及其他相关重要问题提供新的见解。
英文摘要：
The Universal Approximation Theorem posits that neural networks can theoretically possess unlimited approximation capacity with a suitable activation function and a freely chosen or trained set of parameters. However, a more practical scenario arises when these neural parameters, especially the nonlinear weights and biases, are bounded. This leads us to question: \textbf{Does the approximation capacity of a neural network remain universal, or does it have a limit when the parameters are practically bounded? And if it has a limit, how can it be measured?} Our theoretical study indicates that while universal approximation is theoretically feasible, in practical numerical scenarios, Deep Neural Networks (DNNs) with any analytic activation functions (such as Tanh and Sigmoid) can only be approximated by a finite-dimensional vector space under a bounded nonlinear parameter space (NP space), whether in a continuous or discrete sense. Based on this study, we introduce the concepts of \textit{$ε$ outer measure} and \textit{Numerical Span Dimension (NSdim)} to quantify the approximation capacity limit of a family of networks both theoretically and practically. Furthermore, drawing on our new theoretical study and adopting a fresh perspective, we strive to understand the relationship between back-propagation neural networks and random parameter networks (such as the Extreme Learning Machine (ELM)) with both finite and infinite width. We also aim to provide fresh insights into regularization, the trade-off between width and depth, parameter space, width redundancy, condensation, and other related important issues.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
