arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2306.04815cs.LGmath.OCstat.ML
收藏

SGD中的弹射现象:训练损失尖峰及其通过特征学习对泛化性能的影响

Catapults in SGD: spikes in the training loss and their impact on generalization through feature learning

UC San Diego(加州大学圣迭戈分校);Halicioğlu Data Science Institute(哈利奇奥卢数据科学研究所);Harvard University(哈佛大学);Broad Institute of MIT and Harvard(麻省理工学院与哈佛大学布罗德研究所)

Libin Zhu, Chaoyue Liu, Adityanarayanan Radhakrishnan, Mikhail Belkin

AI总结:

本文揭示SGD训练损失尖峰本质上是低维子空间中的弹射现象,并证明弹射通过增强与AGOP的对齐促进特征学习,且小批量SGD能诱发更多弹射从而提升泛化性能。

摘要:

本文首先对神经网络使用随机梯度下降(SGD)训练时训练损失中普遍出现的尖峰现象给出解释。我们提供证据表明,SGD训练损失中的尖峰是“弹射”(catapults)现象,该优化现象最初由[Lewkowycz et al. 2020]在大学习率梯度下降(GD)中观察到。我们通过实验证明,无论是GD还是SGD,这些弹射现象都发生在由切向核(tangent kernel)顶部特征向量张成的低维子空间中。其次,我们提出一种解释,说明弹射如何带来更好的泛化性能:我们证明弹射通过增加与真实预测器的平均梯度外积(AGOP)的对齐程度来促进特征学习。此外,我们证明SGD中较小的批量大小会诱发更多次弹射,从而改善AGOP对齐和测试性能。

英文摘要:

In this paper, we first present an explanation regarding the common occurrence of spikes in the training loss when neural networks are trained with stochastic gradient descent (SGD). We provide evidence that the spikes in the training loss of SGD are "catapults", an optimization phenomenon originally observed in GD with large learning rates in [Lewkowycz et al. 2020]. We empirically show that these catapults occur in a low-dimensional subspace spanned by the top eigenvectors of the tangent kernel, for both GD and SGD. Second, we posit an explanation for how catapults lead to better generalization by demonstrating that catapults promote feature learning by increasing alignment with the Average Gradient Outer Product (AGOP) of the true predictor. Furthermore, we demonstrate that a smaller batch size in SGD induces a larger number of catapults, thereby improving AGOP alignment and test performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑