arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2302.07011cs.LG
收藏

锐度与泛化之间关系的现代审视

A Modern Look at the Relationship between Sharpness and Generalization

EPFL(洛桑联邦理工学院);Tübingen AI Center(图宾根人工智能中心);University of Tübingen(图宾根大学)

Maksym Andriushchenko, Francesco Croce, Maximilian Müller, Matthias Hein, Nicolas Flammarion

AI总结:

本文系统评估多种 adaptive sharpness 在 ImageNet、CIFAR-10、CLIP 和 BERT 等现代训练与微调场景中的表现,发现锐度并不能稳定预测泛化,反而常受学习率等因素影响。

摘要:

极小值的锐度是一个很有前景的量,它可能与深度网络的泛化能力相关,并且在训练过程中对其进行优化时,可以改善泛化。然而,标准锐度在神经网络的重参数化下并不具有不变性;为解决这一问题,研究者提出了重参数化不变的锐度定义,其中最突出的是 adaptive sharpness(Kwon et al., 2021)。但它在现代实际场景中是否真正刻画了泛化?我们在一项详细研究中全面探讨了这一问题,考察多种 adaptive sharpness 定义,场景涵盖在 ImageNet 和 CIFAR-10 上从头训练,以及在 ImageNet 上微调 CLIP、在 MNLI 上微调 BERT。我们主要关注 transformer;尽管 transformer 已被广泛使用,但人们对其锐度所知甚少。总体而言,我们观察到锐度与泛化的相关性并不好,反而与学习率等某些训练参数相关,而这些参数根据具体设置可能与泛化呈正相关或负相关。有趣的是,在多种情况下,我们观察到锐度与分布外误差存在一致的负相关,这意味着更尖锐的极小值可能泛化得更好。最后,我们在一个简单模型上说明,正确的锐度度量高度依赖于数据,而对于现实数据分布,我们对这一方面仍缺乏充分理解。我们实验的代码可在 https://github.com/tml-epfl/sharpness-vs-generalization 获取。

英文摘要:

Sharpness of minima is a promising quantity that can correlate with generalization in deep networks and, when optimized during training, can improve generalization. However, standard sharpness is not invariant under reparametrizations of neural networks, and, to fix this, reparametrization-invariant sharpness definitions have been proposed, most prominently adaptive sharpness (Kwon et al., 2021). But does it really capture generalization in modern practical settings? We comprehensively explore this question in a detailed study of various definitions of adaptive sharpness in settings ranging from training from scratch on ImageNet and CIFAR-10 to fine-tuning CLIP on ImageNet and BERT on MNLI. We focus mostly on transformers for which little is known in terms of sharpness despite their widespread usage. Overall, we observe that sharpness does not correlate well with generalization but rather with some training parameters like the learning rate that can be positively or negatively correlated with generalization depending on the setup. Interestingly, in multiple cases, we observe a consistent negative correlation of sharpness with out-of-distribution error implying that sharper minima can generalize better. Finally, we illustrate on a simple model that the right sharpness measure is highly data-dependent, and that we do not understand well this aspect for realistic data distributions. The code of our experiments is available at https://github.com/tml-epfl/sharpness-vs-generalization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑