发表机构
Unité de Mathématiques Pures et Appliquées (UMPA), ENS Lyon; CNRS（纯数学与应用数学单位（UMPA），里昂高等师范学校; 法国国家科学研究中心）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究高维中学习错误指定单指标模型的无限宽两层神经网络总体梯度流，通过摄动参数联合优化两层，证明隐藏链接函数相关分量能在预测时间尺度恢复，分析二次分量学习起始及权重经验测度奇异行为。
AI 中文摘要
我们研究了在高维中学习错误指定的单指标模型的无限宽两层神经网络的总体梯度流。两层联合优化，有一个摄动参数调整第一层和第二层之间的相对训练速度。Berthier、Montanari和Zhou曾考虑过此设置并猜想了一种分层学习场景。本文证明了隐藏链接函数的常数和线性分量确实能在预测时间尺度内、在精确的显式阈值下恢复。接着分析了二次分量学习的起始，并表明早期学习的分量继续以重要方式影响动力学。证明基于奇异摄动流在由积分约束定义的流形附近演化的定量逼近结果。在现象学层面，我们还表明权重的经验测度在达到隐藏链接的二次分量时显示出奇异行为，一小部分神经元显著增长，其余神经元重新排列以保留已学习的分量。
英文摘要
We study the population gradient flow of an infinitely wide two-layer neural network learning a misspecified single-index model in high dimension. The two layers are optimized jointly, with a perturbative parameter tuning the relative training speed between the first and second layer. This setting was considered by Berthier, Montanari and Zhou in \cite{berthier2024learning}, who conjectured a hierarchical learning scenario with explicit timescales as the second layer is trained faster than the first. In this paper, we prove that the constant and linear components of the hidden link function are indeed recovered within the predicted timescales, at sharp explicit thresholds. We then analyze the onset of learning of the quadratic component and show that the components learned at earlier stages continue to influence the dynamics in an essential way. Our proof is based on quantitative approximation results for singularly perturbed flows evolving near a manifold defined by integral constraints. At a phenomenological level, we also show that the empirical measure of the weights displays singular behaviour when reaching the quadratic component of the hidden link, with a small fraction of neurons growing significantly while the remaining ones rearrange to preserve the components already learned.
Comments37 pages
