归一化层是锐度感知最小化所需的全部
Normalization Layers Are All That Sharpness-Aware Minimization Needs
AI总结：
研究发现SAM仅扰动归一化层参数即可超越全参数扰动，此现象跨模型与变体成立，且非其他稀疏扰动可复制，质疑锐度降低为唯一机制。
摘要：
锐度感知最小化（SAM）被提出用于降低最小值的锐度，并已在多种设置中显示出增强泛化性能的效果。在本工作中，我们表明在SAM的对抗步骤中仅扰动仿射归一化参数（通常占总参数的0.1%）可以优于扰动所有参数。这一发现推广到不同的SAM变体以及ResNet（批归一化）和Vision Transformer（层归一化）架构。我们考虑了替代的稀疏扰动方法，并发现这些方法在如此极端的稀疏水平下无法实现类似的性能提升，表明这种行为是归一化层所独有的。尽管我们的发现重申了SAM在提高泛化性能方面的有效性，但它们对这是否完全由锐度降低所致提出了质疑。
英文摘要：
Sharpness-aware minimization (SAM) was proposed to reduce sharpness of minima and has been shown to enhance generalization performance in various settings. In this work we show that perturbing only the affine normalization parameters (typically comprising 0.1% of the total parameters) in the adversarial step of SAM can outperform perturbing all of the parameters.This finding generalizes to different SAM variants and both ResNet (Batch Normalization) and Vision Transformer (Layer Normalization) architectures. We consider alternative sparse perturbation approaches and find that these do not achieve similar performance enhancement at such extreme sparsity levels, showing that this behaviour is unique to the normalization layers. Although our findings reaffirm the effectiveness of SAM in improving generalization performance, they cast doubt on whether this is solely caused by reduced sharpness.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
