稳定性边界上梯度下降的多时间尺度：中心流的微扰推导

中文摘要 AI 辅助

Cohen等人（2025）提出的中心流是深度学习中稳定性边界处梯度下降的经验准确连续时间模型，但其推导具有启发性。我们提出一种微扰机制，其中中心流是梯度下降的极限：假设损失函数分解为$f = g + \boldsymbol{\text{ε}} h$；在$\boldsymbol{\text{ε}} \to 0$的极限下，学习率为$\boldsymbol{\text{η}}$的梯度下降动力学收敛于$h$的梯度流，且该梯度流被约束在$g$的极小值点上，其尖锐度至多为$2/\boldsymbol{\text{η}}$。我们的方法是形式化的而非严格的，将梯度下降视为$\boldsymbol{\text{ε}}$中的奇异摄动动力系统。研究中出现三个时间尺度：沿最尖锐方向振荡的快时间尺度、自稳定机制的中间时间尺度，以及沿$g$极小值点动力学的慢时间尺度——即中心流。利用奇异摄动理论中的经典形式化方法多尺度方法，我们推导了动力学在$\boldsymbol{\text{ε}}$中的展开式：中心流是展开中的主导项，而自稳定机制出现在下一阶项。我们在已有分析之外研究该机制：当稳定性边界处有单个特征值时，计算波动能量的慢漂移；当稳定性边界处有多个特征值时，推导自稳定系统并解释波动持续存在的原因。

英文摘要

The central flow of Cohen et al. (2025) is an empirically accurate continuous-time model of gradient descent at the edge of stability in deep learning, However, its derivation is heuristic. We propose a perturbative regime in which the central flow is the limit of gradient descent: we assume that the loss decomposes as $f = g + \varepsilon h$; in the limit $\varepsilon \to 0$, the dynamics of gradient descent with learning rate $η$ converge to the gradient flow of $h$ constrained to the minimizers of $g$ of sharpness at most $2/η$. Our approach is formal rather than rigorous; it treats gradient descent as a singularly perturbed dynamical system in $\varepsilon$. Three timescales emerge: a fast timescale of oscillations along the sharpest direction, an intermediate timescale of the self-stabilization mechanism, and a slow timescale of the dynamics along the minimizers of $g$-the central flow. Using the method of multiple scales, a classical formal method from singular perturbation theory, we derive the expansion of the dynamics in $\varepsilon$: the central flow emerges as the leading-order term in the expansion, while the self-stabilization mechanism appears in the next-order term. We study this mechanism beyond previous analyses: with a single eigenvalue at the edge of stability, we compute the slow drift of the energy of the fluctuations; with several eigenvalues at the edge of stability, we derive the self-stabilization system and explain why fluctuations persist.

发表机构

• Sorbonne Université（索邦大学）
• Inria（法国国家信息与自动化研究所）
• Centre Inria de Sorbonne Université（索邦大学法国国家信息与自动化中心）

机构由 AI 辅助整理，请以论文原文为准。