arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2607.02003stat.MLcs.LG
收藏

天生离散,后天平滑:浅层神经网络的变分公式化

Born Discrete, Made Smooth: Variational Formulation of Shallow Neural Networks

Institute of Mathematics, Faculty of Mechanical Engineering, Brno University of Technology(布拉格技术大学数学系,机械工程系);Université de Toulouse, INSA Toulouse, CNRS, IMT(图卢兹大学,图卢兹INSA,CNRS,IMT);University of Warsaw(华沙大学)

Matej Benko, Pierre Bousquet, Iwona Chlebicka, Błażej Miasojedow

AI总结:

提出用连续变分替代浅层神经网络的离散训练问题,证明在加权Sobolev空间中存在λ-凸泛函,通过求解单个线性系统获得最优参数密度,并建立显式泛化误差控制。

摘要:

尽管神经网络非常有效,但其底层优化原理在理论上仍然难以捉摸,通常表现为非凸景观和随机启发式方法。在这项工作中,我们提出了一种范式转变,用适定的连续变分替代浅层神经网络的离散训练问题。我们在加权Sobolev空间中的参数密度上识别出一族$\lambda$-凸泛函,并证明这些变分问题是全局适定、稳定的,并表现出意想不到的几乎$C^3$正则性。与现有的基于Wasserstein或平均场的方法(通常面临有限的正则性和离散化挑战)不同,我们的公式直接提供了椭圆正则性和凸分析。这使我们能够证明最优参数密度可以通过求解单个线性系统获得,完全绕过迭代优化。我们建立了相对于正则化参数以$1/\alpha$速率的显式泛化误差控制,并证明宽度为$N$的有限宽度网络以$O(1/N)$速率达到连续最优。这种观点弥合了神经正切核(NTK)和特征学习机制之间的差距,为通过变分法理解过参数化提供了一个原则性框架。

英文摘要:

Although neural networks are remarkably effective, their underlying optimization principles remain theoretically elusive, often characterized by non-convex landscapes and stochastic heuristics. In this work, we propose a paradigm shift by replacing the discrete training problem of shallow neural networks with a well-posed continuum variational surrogate. We identify a family of $λ$-convex functionals over parameter densities in weighted Sobolev spaces and prove that these variational problems are globally well-posed, stable, and exhibit unexpected almost $C^3$ regularity. Unlike existing Wasserstein-based or Mean-Field approaches, which often face limited regularity and discretization challenges, our formulation provides direct access to elliptic regularity and convex analysis. This allows us to prove that the optimal parameter density can be obtained by solving a single linear system, bypassing iterative optimization entirely. We establish explicit generalization error controls at a rate of $1/α$ relative to the regularization parameter, and prove that finite-width networks of size $N$ achieve the continuum optimum at an $O(1/N)$ rate. This perspective bridges the gap between the Neural Tangent Kernel (NTK) and feature-learning regimes, providing a principled framework for understanding over-parameterization through the lens of variational calculus.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑