arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.20892math.OCmath.PR
收藏

通过均场公式解释大神经网络的隐含正则化

Implicit Regularization of Large Neural Networks via Mean-Field Formulation

ETH Zürich(苏黎世联邦理工学院);UC Berkeley(加州大学伯克利分校);TU Graz(格拉茨工业大学);Princeton University(普林斯顿大学)

Beatrice Acciaio, Jakob Heiss, Gudmund Pammer, Qinxin Yan

AI总结:

本文提出一个数学框架,解释过参数化神经网络训练中早停现象的隐含正则化,通过均场极限下的概率测度梯度流,建立马尔可夫控制理论,并定义新的概率测度度量,揭示早停选择的正则化项。

摘要:

我们提出一个数学框架,以解释过参数化神经网络训练中早停现象的隐含正则化。在均场极限下,参数分布按照概率测度空间上的梯度流演化。我们展示这些动态可通过对应的哈密顿-雅可比-贝尔曼(HJB)方程转化为马尔可夫控制理论。控制观点提供了动态规划原理(DPP),我们利用其定义概率测度上的新度量。该度量可视为均场环境下控制表示的Wasserstein-2距离推广,并自然出现在早停选择的正则化项中。我们进一步获得非渐进行为界,描述诱导正则化如何依赖于停止时间。

英文摘要:

We propose a mathematical framework to explain implicit regularization from early stopping during the training of overparametrized neural networks. In the mean-field limit, the parameter distribution evolves according to a gradient flow on the space of probability measures. We show that these dynamics admit an equivalent McKean-Vlasov stochastic control formulation through the corresponding Hamilton-Jacobi-Bellman (HJB) equation. The control viewpoint yields a Dynamic Programming Principle (DPP), which we use to define a new metric on probability measures. This metric can be viewed as a mean-field generalization of the control representation of the Wasserstein-2 distance, and it naturally appears as a regularization term selected by early stopping. We further obtain non-asymptotic bounds describing how the induced regularization depends on the stopping time.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑