在平坦极小值流形附近大学习率梯度下降的动力学

中文摘要 AI 辅助

梯度下降理论中的一个重要量是“锐度”，定义为目标函数海森矩阵的最大特征值。经典分析通常要求步长均匀小于锐度倒数的两倍，但在深度神经网络训练中此条件常被违反。近期工作在单标量输出的超参数化最小二乘设置中弥补了这一差距。本文将该理论扩展到向量值输出的超参数化最小二乘（包括任意多观测值的回归）以及平坦极小值流形的邻域。我们将相关范式和收敛定理推广到更广泛的设置，克服了几个技术挑战，包括通过一种可能具有独立意义的新方法解决奇异偏微分方程。我们还表明该框架在温和假设下适用于深度矩阵分解，得出了几个新的结构结果。

英文摘要

An important quantity in the theory of gradient descent (GD) is the \emph{sharpness}, defined as the largest eigenvalue of the objective Hessian. Classical analyses typically require the step size to be uniformly smaller than twice the reciprocal of the sharpness, but this condition is frequently violated in the training of deep neural networks. Recent work bridges this gap in the setting of overparametrised least-squares with a \emph{single scalar output}, providing a normal form for large-step GD in a neighbourhood of an \emph{isolated} flat minimum and establishing three corresponding convergence results. In this paper, we extend this theory in two directions: (1) to overparametrised least-squares with \emph{vector-valued outputs} (including regression with arbitrarily many observations), and (2) to a neighbourhood of a \emph{manifold} of flat minima (which we show is essential for applications such as matrix factorisation). We generalise both the normal form and all three convergence theorems of \cite{macdonaldeos} to this broader setting, overcoming several technical challenges. We further show that our framework applies to deep matrix factorisation under mild assumptions, yielding several new structural results. In particular, we prove that the set of flat minima forms a fibre bundle over a product of spheres, and that the sharpness is Morse-Bott along this manifold.

发表机构

• University of Pennsylvania（宾夕法尼亚大学）

机构由 AI 辅助整理，请以论文原文为准。