首页/论文/2006.09355

arXiv 2006.093552020-06-16cs.LGmath.OCmath.STstat.MLstat.TH

A Note on the Global Convergence of Multilayer Neural Networks in the Mean Field Regime

Stanford University（斯坦福大学）

Huy Tuan Pham, Phan-Minh Nguyen

备注：: Companion note to arXiv:2001.11443

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

英文摘要相关论文

英文摘要：

In a recent work, we introduced a rigorous framework to describe the mean field limit of the gradient-based learning dynamics of multilayer neural networks, based on the idea of a neuronal embedding. There we also proved a global convergence guarantee for three-layer (as well as two-layer) networks using this framework. In this companion note, we point out that the insights in our previous work can be readily extended to prove a global convergence guarantee for multilayer networks of any depths. Unlike our previous three-layer global convergence guarantee that assumes i.i.d. initializations, our present result applies to a type of correlated initialization. This initialization allows to, at any finite training time, propagate a certain universal approximation property through the depth of the neural network. To achieve this effect, we introduce a bidirectional diversity condition.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

A Rigorous Framework for the Mean Field Limit of Multilayer Neural Networks

The Voleon Group（Voleon集团）；Stanford University（斯坦福大学）；University of Cambridge（剑桥大学）

We develop a mathematically rigorous framework for multilayer neural networks in the mean field regime. As the network's widths increase, the network's learning trajectory is shown to be well captured by a meaningful and dynamically nonlinear limit (the \textit{mean field} limit), which is characterized by a system of ODEs. Our framework applies to a broad range of network architectures, learning dynamics and network initializations. Central to the framework is the new idea of a \textit{neuronal embedding}, which comprises of a non-evolving probability space that allows to embed neural network…

2023-02-13cond-mat.stat-mechcs.LGmath.PR

A Short Note of PAGE: Optimal Convergence Rates for Nonconvex Optimization

KAUST（阿卜杜拉国王科技大学）

In this note, we first recall the nonconvex problem setting and introduce the optimal PAGE algorithm (Li et al., ICML'21). Then we provide a simple and clean convergence analysis of PAGE for achieving optimal convergence rates. Moreover, PAGE and its analysis can be easily adopted and generalized to other works. We hope that this note provides the insights and is helpful for future works.

2021-06-17cs.AIcs.DScs.LG

A Note on the Convergence of Mirrored Stein Variational Gradient Descent under $(L_0,L_1)-$Smoothness Condition

King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；KAUST（阿卜杜拉国王科技大学）

In this note, we establish a descent lemma for the population limit Mirrored Stein Variational Gradient Method~(MSVGD). This descent lemma does not rely on the path information of MSVGD but rather on a simple assumption for the mirrored distribution $\nablaΨ_{\#}π\propto\exp(-V)$. Our analysis demonstrates that MSVGD can be applied to a broader class of constrained sampling problems with non-smooth $V$. We also investigate the complexity of the population limit MSVGD in terms of dimension $d$.

2022-06-20cs.LGmath.OCmath.ST

关于Wasserstein策略优化收敛性的注记

A note on convergence of Wasserstein policy optimization

School of Mathematics, University of Edinburgh（爱丁堡大学数学学院）；Department of Mathematics, Imperial College London（伦敦帝国理工学院数学系）

本文探讨了Wasserstein策略优化在连续状态和动作空间中的收敛性问题，通过利用均场分析和log-Sobole不等式，证明了在熵正则化的马尔可夫决策过程框架下，WPO算法能够线性收敛到全局最优解。

2026-05-21cs.LGmath.OC

一种基于模型平均的多层网络迁移学习框架

A Transfer Learning Framework for Multilayer Networks via Model Averaging

International Institute of Finance, School of Management, University of Science and Technology of China（国际金融研究院，管理学院，中国科学技术大学）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（数学系统科学研究院，中国科学院）

针对现有多层网络链路预测方法依赖共享结构假设、需原始辅助数据的问题，提出基于双层模型平均的迁移学习框架，通过边的K折交叉验证自动加权层间层内模型，兼具理论最优性、高效隐私性，在仿真和真实推荐系统中表现更优。

2025-06-14cs.LGstat.MEstat.ML

MLPGradientFlow: going with the flow of multilayer perceptrons (and finding minima fast and accurately)

MLPGradientFlow is a software package to solve numerically the gradient flow differential equation $\dot θ= -\nabla \mathcal L(θ; \mathcal D)$, where $θ$ are the parameters of a multi-layer perceptron, $\mathcal D$ is some data set, and $\nabla \mathcal L$ is the gradient of a loss function. We show numerically that adaptive first- or higher-order integration methods based on Runge-Kutta schemes have better accuracy and convergence speed than gradient descent with the Adam optimizer. However, we find Newton's method and approximations like BFGS preferable to find fixed points (local and global…

2023-01-25cs.LG

更多相关论文收起更多论文

A note on the complex and bicomplex valued neural networks

Chapman University（查普曼大学）

In this paper we first write a proof of the perceptron convergence algorithm for the complex multivalued neural networks (CMVNNs). Our primary goal is to formulate and prove the perceptron convergence algorithm for the bicomplex multivalued neural networks (BMVNNs) and other important results in the theory of neural networks based on a bicomplex algebra.

2022-02-04cs.ITcs.LGmath.IT

关于去噪扩散概率模型收敛性的一则注记

A Note on the Convergence of Denoising Diffusion Probabilistic Models

Université Laval（拉瓦尔大学）；University College London（伦敦大学学院）

该研究针对扩散模型收敛性问题，基于Mbacke等人2023年的工作，推导了数据生成分布与扩散模型学习分布间Wasserstein距离的定量上界，无需对得分函数假设，适用于有界空间任意分布且无指数依赖，证明初等。

2024-09-15cs.LG

学习正确的层：多层图上用于半监督学习的数据驱动层聚合策略

Learning the Right Layers: a Data-Driven Layer-Aggregation Strategy for Semi-Supervised Learning on Multilayer Graphs

University of Padova（帕多瓦大学）；University of Rome “Tor Vergata”（罗马第二大学（托尔·韦尔加塔））；Gran Sasso Science Institute（格兰萨索科学研究所）

针对多层图半监督学习中各层贡献难以先验评估的问题，本文提出一种无参数拉普拉斯正则化模型，通过Frank-Wolfe优化与改进标签传播从少量标签中学习层的最优非线性组合，实验证明其优于各层单独使用及多种基线方法。

2023-05-31cs.LG

Interaction Matters: A Note on Non-asymptotic Local Convergence of Generative Adversarial Networks

University of Chicago, Booth School of Business（芝加哥大学布斯商学院）；University of Pennsylvania（宾夕法尼亚大学）

Motivated by the pursuit of a systematic computational and algorithmic understanding of Generative Adversarial Networks (GANs), we present a simple yet unified non-asymptotic local convergence theory for smooth two-player games, which subsumes several discrete-time gradient-based saddle point dynamics. The analysis reveals the surprising nature of the off-diagonal interaction term as both a blessing and a curse. On the one hand, this interaction term explains the origin of the slow-down effect in the convergence of Simultaneous Gradient Ascent (SGA) to stable Nash equilibria. On the other hand…

2019-02-23cs.GTcs.LGstat.ML

A Note on Inexact Condition for Cubic Regularized Newton's Method

Ohio State University（俄亥俄州立大学）；Georgia Institute of Technology（佐治亚理工学院）

This note considers the inexact cubic-regularized Newton's method (CR), which has been shown in \cite{Cartis2011a} to achieve the same order-level convergence rate to a secondary stationary point as the exact CR \citep{Nesterov2006}. However, the inexactness condition in \cite{Cartis2011a} is not implementable due to its dependence on future iterates variable. This note fixes such an issue by proving the same convergence rate for nonconvex optimization under an inexact adaptive condition that depends on only the current iterate. Our proof controls the sufficient decrease of the function value …

2018-08-22cs.LGmath.OC

多层神经网络中随机梯度下降的相图与特征值动态

Phase diagram and eigenvalue dynamics of stochastic gradient descent in multilayer neural networks

Centre for Quantum Fields and Gravity（量子场与引力中心）；Department of Physics, Swansea University（斯旺西大学物理系）；School of Mathematical Sciences, Queen Mary University of London（伦敦大学量子玛丽学院数学科学学院）

研究通过分析多层神经网络的相图，揭示随机梯度下降在不同训练阶段的动力学特征，为超参数选择提供理论依据。

2025-11-18cond-mat.dis-nncs.LGhep-lat

无监督神经网络求解反问题的收敛性与恢复保证

Convergence and Recovery Guarantees of Unsupervised Neural Networks for Inverse Problems

Normandie Univ.（诺曼底大学）；UNICAEN（卡昂大学）；ENSICAEN（卡昂高等工程师学院）；CNRS（法国国家科学研究中心）；Greyc（格雷西实验室）

本研究为无监督前馈多层神经网络求解反问题提供确定性收敛与恢复保证，并推导出两层深度逆先验网络适用的过参数化界限。

2024-03-15cs.LG