所走之路：优化器在稳定性边界处的作用
The Road Taken: The Role of Optimizers at the Edge of Stability
- Seoul National University（首尔大学）
- Computer Vision Lab, ASRI（ASRI计算机视觉实验室）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
该研究针对深度学习中稳定性边界的过往公式未捕捉优化器依赖性的问题，推导了基于定向海森矩阵与梯度对齐得分的新稳定性边界公式，揭示了优化器平衡时间与空间预算的独特作用。
AI中文摘要：
稳定性边界是基于梯度的优化器在深度学习中出现的一种现象，即损失函数的海森矩阵特征值保持稳定，其阈值高于经典下降引理所预测的不稳定阈值。过往研究针对最大海森特征值和学习率构建了稳定性边界的相关公式，但我们发现，包括梯度下降在内的多种一阶方法，对这些理论预测的稳定性边界的违反程度高达21.1倍，且这种偏差是系统性的，高度依赖于底层优化器，而过往公式未捕捉到这一点。这需要构建新的稳定性边界公式，我们从定向海森矩阵和优化器实际采用的更新对应的梯度对齐得分出发，而非最大曲率模式，推导得出了该公式。我们提出的已实现稳定性边界的新公式，不仅消除了优化器相关的偏移，对稳定性阈值提供了更一致的预测，还引入了新的诊断工具，揭示了优化器在一阶优化中主动平衡时间与空间预算的独特作用。
英文摘要：
The edge of stability refers to a phenomenon in deep learning with gradient-based optimizers where the Hessian eigenvalues of the loss remain stable above a threshold that the classical descent lemma predicts to be unstable. Previous works formulate the edge of stability with respect to the maximum Hessian eigenvalue and the learning rate. However, we observe that many first-order methods, including gradient descent, significantly violate the stability bound predicted by these theories by a factor as large as $\times 21.1$. Moreover, this deviation turns out to be systematic and highly dependent on the underlying optimizer, which is not captured by previous formulations. This calls for a new formulation of the stability threshold, which we derive from the directional Hessian and the gradient-alignment score with respect to the actual update taken by the optimizer, rather than the maximum curvature mode. Our new formulation of the realized edge of stability not only removes optimizer-dependent offsets and provides more consistent predictions of the stability threshold, but also introduces new diagnostic tools that reveal the unique role of the optimizer in actively balancing between the temporal and spatial budgets in first-order optimization.
