对角线性网络上的(S)GD：隐式正则化、大步长与稳定性边缘
(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stability
AI总结：
本文研究了随机性与大步长对对角线性网络上GD和SGD隐式正则化的影响，证明其收敛性并通过隐式正则化刻画解，发现大步长有利于SGD稀疏回归但阻碍GD，且该效应在稳定性边缘被放大。
摘要：
本文研究了随机性与大步长对梯度下降（GD）和随机梯度下降（SGD）在对角线性网络上隐式正则化的影响。我们在过参数化回归设置中证明了具有宏观步长的GD和SGD的收敛性，并通过隐式正则化问题刻画了它们的解。我们清晰的刻画得出了关于随机性与步长对恢复解影响的定性见解。具体而言，我们证明了大步长在稀疏回归问题中始终有利于SGD，而它们可能会阻碍GD恢复稀疏解。这些效应在恰好低于发散阈值的狭窄窗口内，即在“稳定性边缘”机制下被放大。我们的发现得到了实验结果的支持。
英文摘要：
In this paper, we investigate the impact of stochasticity and large stepsizes on the implicit regularisation of gradient descent (GD) and stochastic gradient descent (SGD) over diagonal linear networks. We prove the convergence of GD and SGD with macroscopic stepsizes in an overparametrised regression setting and characterise their solutions through an implicit regularisation problem. Our crisp characterisation leads to qualitative insights about the impact of stochasticity and stepsizes on the recovered solution. Specifically, we show that large stepsizes consistently benefit SGD for sparse regression problems, while they can hinder the recovery of sparse solutions for GD. These effects are magnified for stepsizes in a tight window just below the divergence threshold, in the "edge of stability" regime. Our findings are supported by experimental results.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
