神经网络逼近中的函数等价性和几何多样性：实证表征
Functional Equivalence and Geometric Diversity in Neural Network Approximations: An Empirical Characterization
AI总结：
研究神经网络逼近的函数等价性与几何多样性，通过分析单层与多层感知器在有无噪声下的情况，从松散度视角研究几何性质，揭示相关等价类网络特点，还提出基于简约性等的模型选择标准。
AI中文摘要：
通用逼近定理表明，具有单个隐藏层的神经网络足以将紧致域上的任何连续单变量函数逼近到任意误差。然而，这种神经网络表示的唯一性无法保证，引发了关于实际可识别性的问题。在这项工作中，我们通过分析神经网络对一些基本数学函数逼近的函数等价性和几何多样性来解决这一问题。分析包括在有噪声和无噪声条件下对单层神经网络和多层感知器的广泛研究。除了网络容量，我们还通过以代价函数海森矩阵的特征谱和有效秩来量化参数空间维度的松散度视角研究几何性质。研究揭示了功能上不可区分但几何上多样的大型等价类网络，这些网络始终表现出低有效秩和结构冗余。最后，提出了一种基于简约性、估计容易性和推理效率来识别最优模型的模型选择标准。
英文摘要：
The Universal Approximation Theorem states that a neural network with a single hidden layer is sufficient to approximate any continuous univariate function on a compact domain to arbitrary error. However, the uniqueness of such neural network representations is not guaranteed, raising questions about practical identifiability. In this work, we address this concern by analyzing functional equivalence and geometric diversity of neural network approximations to a few elementary mathematical functions. The analysis includes an extensive study of single-layer neural networks and multilayer perceptrons under noisy and noise-free conditions. Beyond just network capacity, we study the geometric properties through the lens of sloppiness, characterized by the eigen spectrum of the Hessian of the cost function and the effective rank to quantify the dimensionality of parameter space. The study reveals large equivalence classes of functionally indistinguishable yet geometrically diverse networks that consistently exhibit low effective rank and structural redundancy. Finally, a model select criterion is proposed for identifying optimal models based on parsimony, ease of estimation, and inference efficiency.
